2024-04-19T01:29:30.046 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-19T01:29:30.046 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663309 --name yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi --description rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/pool-create-delete} --owner scheduled_yuriw@teuthology --verbose -- /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663309/orig.config.yaml 2024-04-19T01:29:30.089 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-19T01:29:30.169 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663309 branch: wip-yuri5-testing-2024-04-17-1400 description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/pool-create-delete} email: yweinste@redhat.com first_in_suite: false job_id: '7663309' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-yuri5-testing-2024-04-17-1400 ceph: conf: client: debug monc: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: mon ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos min: 10 paxos service trim min: 5 paxos trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(POOL_APP_NOT_ENABLED\) - slow request - overall HEALTH_ sha1: 8b4ab94f9c30da63487fa2f241ce12f483378860 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 8b4ab94f9c30da63487fa2f241ce12f483378860 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri5-testing-2024-04-17-1400 sha1: 8b4ab94f9c30da63487fa2f241ce12f483378860 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 1937 sha1: 8b4ab94f9c30da63487fa2f241ce12f483378860 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri5-testing-2024-04-17-1400 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_8b4ab94f9c30da63487fa2f241ce12f483378860/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 8b4ab94f9c30da63487fa2f241ce12f483378860 targets: smithi062.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGrkx9A8skJaBcqTAomFkrqQVf4BA59s75KrIp7KfodFGwB3b4ArrQb9AWLRBdB4PVe1pxu6vFSUKjbhUjuWZ8M= smithi071.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEHCgE5GNnmLfQFcqynfoGEh+GOgSEJ8VGUFw7je1VxDrEUx1PPhOwpgb11xb3ZmToGPmFF3uU0FRm3xn29h0aM= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 90 thrash_delay: 1 thrash_many: true - exec: client.0: - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-18_22:49:05 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-19T01:29:30.169 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_8b4ab94f9c30da63487fa2f241ce12f483378860/qa; will attempt to use it 2024-04-19T01:29:30.169 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_8b4ab94f9c30da63487fa2f241ce12f483378860/qa/tasks 2024-04-19T01:29:30.170 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-19T01:29:30.171 INFO:teuthology.task.internal:Checking packages... 2024-04-19T01:29:30.193 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '8b4ab94f9c30da63487fa2f241ce12f483378860' 2024-04-19T01:29:30.193 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-19T01:29:30.193 INFO:teuthology.packaging:ref: None 2024-04-19T01:29:30.193 INFO:teuthology.packaging:tag: None 2024-04-19T01:29:30.193 INFO:teuthology.packaging:branch: wip-yuri5-testing-2024-04-17-1400 2024-04-19T01:29:30.193 INFO:teuthology.packaging:sha1: 8b4ab94f9c30da63487fa2f241ce12f483378860 2024-04-19T01:29:30.194 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri5-testing-2024-04-17-1400 2024-04-19T01:29:30.411 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T01:29:30.413 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-19T01:29:30.419 INFO:teuthology.task.internal:no buildpackages task found 2024-04-19T01:29:30.419 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-19T01:29:30.426 INFO:teuthology.task.internal:Saving configuration 2024-04-19T01:29:30.444 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-19T01:29:30.451 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-19T01:29:30.472 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi062.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663309', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-19 01:23:56.215585', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKvEhQGlOW3jxP+MQ+NhjfXSS+msXNdePWxQdxfGbrgB'} 2024-04-19T01:29:30.494 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663309', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-19 01:23:56.217123', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIDZBZCUZka2XKkVyVdKQqJ+sl0wtX7A1Se1oiHPbduy'} 2024-04-19T01:29:30.494 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-19T01:29:30.499 INFO:teuthology.task.internal:roles: ubuntu@smithi062.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-04-19T01:29:30.499 INFO:teuthology.task.internal:roles: ubuntu@smithi071.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-04-19T01:29:30.499 INFO:teuthology.run_tasks:Running task console_log... 2024-04-19T01:29:30.561 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3faff0f3a0>, signals=[15]) 2024-04-19T01:29:30.562 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-19T01:29:30.566 INFO:teuthology.task.internal:Opening connections... 2024-04-19T01:29:30.566 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi062.front.sepia.ceph.com 2024-04-19T01:29:30.568 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:29:30.654 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi071.front.sepia.ceph.com 2024-04-19T01:29:30.655 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:29:30.732 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-19T01:29:30.739 DEBUG:teuthology.orchestra.run.smithi062:> uname -m 2024-04-19T01:29:30.745 INFO:teuthology.orchestra.run.smithi062.stdout:x86_64 2024-04-19T01:29:30.745 DEBUG:teuthology.orchestra.run.smithi062:> cat /etc/os-release 2024-04-19T01:29:30.793 INFO:teuthology.orchestra.run.smithi062.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-19T01:29:30.793 INFO:teuthology.orchestra.run.smithi062.stdout:NAME="Ubuntu" 2024-04-19T01:29:30.793 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_ID="22.04" 2024-04-19T01:29:30.793 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-19T01:29:30.793 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_CODENAME=jammy 2024-04-19T01:29:30.793 INFO:teuthology.orchestra.run.smithi062.stdout:ID=ubuntu 2024-04-19T01:29:30.793 INFO:teuthology.orchestra.run.smithi062.stdout:ID_LIKE=debian 2024-04-19T01:29:30.793 INFO:teuthology.orchestra.run.smithi062.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-19T01:29:30.794 INFO:teuthology.orchestra.run.smithi062.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-19T01:29:30.794 INFO:teuthology.orchestra.run.smithi062.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-19T01:29:30.794 INFO:teuthology.orchestra.run.smithi062.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-19T01:29:30.794 INFO:teuthology.orchestra.run.smithi062.stdout:UBUNTU_CODENAME=jammy 2024-04-19T01:29:30.794 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2024-04-19T01:29:30.817 DEBUG:teuthology.orchestra.run.smithi071:> uname -m 2024-04-19T01:29:30.823 INFO:teuthology.orchestra.run.smithi071.stdout:x86_64 2024-04-19T01:29:30.823 DEBUG:teuthology.orchestra.run.smithi071:> cat /etc/os-release 2024-04-19T01:29:30.870 INFO:teuthology.orchestra.run.smithi071.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-19T01:29:30.871 INFO:teuthology.orchestra.run.smithi071.stdout:NAME="Ubuntu" 2024-04-19T01:29:30.871 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION_ID="22.04" 2024-04-19T01:29:30.871 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-19T01:29:30.871 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION_CODENAME=jammy 2024-04-19T01:29:30.871 INFO:teuthology.orchestra.run.smithi071.stdout:ID=ubuntu 2024-04-19T01:29:30.871 INFO:teuthology.orchestra.run.smithi071.stdout:ID_LIKE=debian 2024-04-19T01:29:30.871 INFO:teuthology.orchestra.run.smithi071.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-19T01:29:30.871 INFO:teuthology.orchestra.run.smithi071.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-19T01:29:30.871 INFO:teuthology.orchestra.run.smithi071.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-19T01:29:30.871 INFO:teuthology.orchestra.run.smithi071.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-19T01:29:30.871 INFO:teuthology.orchestra.run.smithi071.stdout:UBUNTU_CODENAME=jammy 2024-04-19T01:29:30.871 INFO:teuthology.lock.ops:Updating smithi071.front.sepia.ceph.com on lock server 2024-04-19T01:29:30.891 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-19T01:29:30.900 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-19T01:29:30.904 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-19T01:29:30.904 DEBUG:teuthology.orchestra.run.smithi062:> test '!' -e /home/ubuntu/cephtest 2024-04-19T01:29:30.907 DEBUG:teuthology.orchestra.run.smithi071:> test '!' -e /home/ubuntu/cephtest 2024-04-19T01:29:30.914 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-19T01:29:30.920 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-19T01:29:30.920 DEBUG:teuthology.orchestra.run.smithi062:> test -z $(ls -A /var/lib/ceph) 2024-04-19T01:29:30.955 DEBUG:teuthology.orchestra.run.smithi071:> test -z $(ls -A /var/lib/ceph) 2024-04-19T01:29:30.979 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-19T01:29:31.032 INFO:teuthology.run_tasks:Running task kernel... 2024-04-19T01:29:31.047 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-19T01:29:31.047 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-19T01:29:31.048 DEBUG:teuthology.orchestra.run.smithi062:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-19T01:29:31.048 DEBUG:teuthology.orchestra.run.smithi071:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-19T01:29:31.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:29:31.053 DEBUG:teuthology.orchestra.run.smithi062:> uname -r 2024-04-19T01:29:31.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:29:31.054 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2024-04-19T01:29:31.100 INFO:teuthology.orchestra.run.smithi071.stdout:5.15.0-56-generic 2024-04-19T01:29:31.100 INFO:teuthology.task.kernel:Running kernel on smithi071: 5.15.0-56-generic 2024-04-19T01:29:31.100 DEBUG:teuthology.orchestra.run.smithi071:> sudo apt-get clean 2024-04-19T01:29:31.102 INFO:teuthology.orchestra.run.smithi062.stdout:5.15.0-56-generic 2024-04-19T01:29:31.102 INFO:teuthology.task.kernel:Running kernel on smithi062: 5.15.0-56-generic 2024-04-19T01:29:31.102 DEBUG:teuthology.orchestra.run.smithi062:> sudo apt-get clean 2024-04-19T01:29:31.225 DEBUG:teuthology.orchestra.run.smithi071:> sudo apt-get update 2024-04-19T01:29:31.240 DEBUG:teuthology.orchestra.run.smithi062:> sudo apt-get update 2024-04-19T01:29:31.498 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T01:29:31.511 INFO:teuthology.orchestra.run.smithi071.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-19T01:29:31.511 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T01:29:31.513 INFO:teuthology.orchestra.run.smithi062.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-19T01:29:31.560 INFO:teuthology.orchestra.run.smithi071.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-19T01:29:31.578 INFO:teuthology.orchestra.run.smithi062.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-19T01:29:31.914 INFO:teuthology.orchestra.run.smithi071.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-19T01:29:31.941 INFO:teuthology.orchestra.run.smithi062.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-19T01:29:31.941 INFO:teuthology.orchestra.run.smithi071.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,351 kB] 2024-04-19T01:29:31.948 INFO:teuthology.orchestra.run.smithi062.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [445 kB] 2024-04-19T01:29:32.054 INFO:teuthology.orchestra.run.smithi071.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,597 kB] 2024-04-19T01:29:32.079 INFO:teuthology.orchestra.run.smithi062.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [616 kB] 2024-04-19T01:29:32.277 INFO:teuthology.orchestra.run.smithi062.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,597 kB] 2024-04-19T01:29:32.310 INFO:teuthology.orchestra.run.smithi062.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,351 kB] 2024-04-19T01:29:32.359 INFO:teuthology.orchestra.run.smithi071.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [445 kB] 2024-04-19T01:29:32.366 INFO:teuthology.orchestra.run.smithi071.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [616 kB] 2024-04-19T01:29:32.376 INFO:teuthology.orchestra.run.smithi071.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [238 kB] 2024-04-19T01:29:32.386 INFO:teuthology.orchestra.run.smithi071.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-19T01:29:32.386 INFO:teuthology.orchestra.run.smithi071.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-19T01:29:32.388 INFO:teuthology.orchestra.run.smithi071.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,680 kB] 2024-04-19T01:29:32.405 INFO:teuthology.orchestra.run.smithi071.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [302 kB] 2024-04-19T01:29:32.420 INFO:teuthology.orchestra.run.smithi071.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-19T01:29:32.420 INFO:teuthology.orchestra.run.smithi071.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,806 kB] 2024-04-19T01:29:32.428 INFO:teuthology.orchestra.run.smithi062.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [302 kB] 2024-04-19T01:29:32.443 INFO:teuthology.orchestra.run.smithi062.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-19T01:29:32.444 INFO:teuthology.orchestra.run.smithi062.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,806 kB] 2024-04-19T01:29:32.479 INFO:teuthology.orchestra.run.smithi071.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [281 kB] 2024-04-19T01:29:32.483 INFO:teuthology.orchestra.run.smithi071.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-19T01:29:32.505 INFO:teuthology.orchestra.run.smithi062.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-19T01:29:32.506 INFO:teuthology.orchestra.run.smithi062.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [305 kB] 2024-04-19T01:29:32.513 INFO:teuthology.orchestra.run.smithi062.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-19T01:29:32.517 INFO:teuthology.orchestra.run.smithi071.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [305 kB] 2024-04-19T01:29:32.517 INFO:teuthology.orchestra.run.smithi071.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-19T01:29:32.518 INFO:teuthology.orchestra.run.smithi071.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-19T01:29:32.523 INFO:teuthology.orchestra.run.smithi071.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-19T01:29:32.533 INFO:teuthology.orchestra.run.smithi071.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [600 kB] 2024-04-19T01:29:32.657 INFO:teuthology.orchestra.run.smithi062.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [238 kB] 2024-04-19T01:29:32.665 INFO:teuthology.orchestra.run.smithi062.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-19T01:29:32.665 INFO:teuthology.orchestra.run.smithi062.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,680 kB] 2024-04-19T01:29:32.679 INFO:teuthology.orchestra.run.smithi062.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-19T01:29:32.687 INFO:teuthology.orchestra.run.smithi071.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-19T01:29:32.709 INFO:teuthology.orchestra.run.smithi071.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-04-19T01:29:32.749 INFO:teuthology.orchestra.run.smithi062.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-19T01:29:32.750 INFO:teuthology.orchestra.run.smithi062.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [281 kB] 2024-04-19T01:29:32.760 INFO:teuthology.orchestra.run.smithi062.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-19T01:29:32.760 INFO:teuthology.orchestra.run.smithi062.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-19T01:29:32.784 INFO:teuthology.orchestra.run.smithi062.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [600 kB] 2024-04-19T01:29:32.961 INFO:teuthology.orchestra.run.smithi062.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-04-19T01:29:33.115 INFO:teuthology.orchestra.run.smithi071.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-19T01:29:33.120 INFO:teuthology.orchestra.run.smithi071.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-19T01:29:33.134 INFO:teuthology.orchestra.run.smithi071.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-19T01:29:33.135 INFO:teuthology.orchestra.run.smithi071.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-19T01:29:33.137 INFO:teuthology.orchestra.run.smithi071.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-19T01:29:33.324 INFO:teuthology.orchestra.run.smithi062.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [699 kB] 2024-04-19T01:29:33.326 INFO:teuthology.orchestra.run.smithi071.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [699 kB] 2024-04-19T01:29:33.355 INFO:teuthology.orchestra.run.smithi062.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-19T01:29:33.362 INFO:teuthology.orchestra.run.smithi062.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-19T01:29:33.362 INFO:teuthology.orchestra.run.smithi062.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-19T01:29:33.363 INFO:teuthology.orchestra.run.smithi071.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-19T01:29:33.375 INFO:teuthology.orchestra.run.smithi062.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-19T01:29:33.377 INFO:teuthology.orchestra.run.smithi071.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-19T01:29:33.377 INFO:teuthology.orchestra.run.smithi062.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-19T01:29:33.377 INFO:teuthology.orchestra.run.smithi062.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-19T01:29:33.378 INFO:teuthology.orchestra.run.smithi071.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-19T01:29:33.378 INFO:teuthology.orchestra.run.smithi071.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-19T01:29:33.379 INFO:teuthology.orchestra.run.smithi062.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-19T01:29:33.380 INFO:teuthology.orchestra.run.smithi062.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-19T01:29:33.380 INFO:teuthology.orchestra.run.smithi062.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-19T01:29:33.380 INFO:teuthology.orchestra.run.smithi071.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-19T01:29:33.381 INFO:teuthology.orchestra.run.smithi071.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-19T01:29:33.381 INFO:teuthology.orchestra.run.smithi071.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-19T01:29:33.397 INFO:teuthology.orchestra.run.smithi062.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-19T01:29:33.399 INFO:teuthology.orchestra.run.smithi062.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-19T01:29:33.401 INFO:teuthology.orchestra.run.smithi062.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-19T01:29:33.406 INFO:teuthology.orchestra.run.smithi062.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-19T01:29:33.407 INFO:teuthology.orchestra.run.smithi071.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-19T01:29:33.410 INFO:teuthology.orchestra.run.smithi062.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-19T01:29:33.411 INFO:teuthology.orchestra.run.smithi071.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-19T01:29:33.575 INFO:teuthology.orchestra.run.smithi071.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-19T01:29:33.576 INFO:teuthology.orchestra.run.smithi062.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-19T01:29:33.657 INFO:teuthology.orchestra.run.smithi062.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-19T01:29:33.658 INFO:teuthology.orchestra.run.smithi071.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-19T01:29:33.744 INFO:teuthology.orchestra.run.smithi062.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-19T01:29:33.777 INFO:teuthology.orchestra.run.smithi071.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-19T01:29:33.822 INFO:teuthology.orchestra.run.smithi062.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-19T01:29:33.824 INFO:teuthology.orchestra.run.smithi071.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-19T01:29:33.857 INFO:teuthology.orchestra.run.smithi071.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-19T01:29:33.860 INFO:teuthology.orchestra.run.smithi062.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-19T01:29:38.087 INFO:teuthology.orchestra.run.smithi071.stdout:Fetched 13.0 MB in 3s (5,076 kB/s) 2024-04-19T01:29:38.574 INFO:teuthology.orchestra.run.smithi062.stdout:Fetched 13.0 MB in 3s (5,121 kB/s) 2024-04-19T01:29:39.387 INFO:teuthology.orchestra.run.smithi071.stdout:Reading package lists... 2024-04-19T01:29:39.405 DEBUG:teuthology.orchestra.run.smithi071:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-19T01:29:39.455 INFO:teuthology.orchestra.run.smithi071.stdout:Reading package lists... 2024-04-19T01:29:39.633 INFO:teuthology.orchestra.run.smithi071.stdout:Building dependency tree... 2024-04-19T01:29:39.633 INFO:teuthology.orchestra.run.smithi071.stdout:Reading state information... 2024-04-19T01:29:39.786 INFO:teuthology.orchestra.run.smithi071.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T01:29:39.786 INFO:teuthology.orchestra.run.smithi071.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T01:29:39.786 INFO:teuthology.orchestra.run.smithi071.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T01:29:39.787 INFO:teuthology.orchestra.run.smithi071.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T01:29:39.787 INFO:teuthology.orchestra.run.smithi071.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-19T01:29:39.787 INFO:teuthology.orchestra.run.smithi071.stdout: python2.7-minimal 2024-04-19T01:29:39.787 INFO:teuthology.orchestra.run.smithi071.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T01:29:39.788 INFO:teuthology.orchestra.run.smithi071.stdout:The following additional packages will be installed: 2024-04-19T01:29:39.788 INFO:teuthology.orchestra.run.smithi071.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-19T01:29:39.788 INFO:teuthology.orchestra.run.smithi071.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-19T01:29:39.789 INFO:teuthology.orchestra.run.smithi071.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-19T01:29:39.790 INFO:teuthology.orchestra.run.smithi071.stdout:Suggested packages: 2024-04-19T01:29:39.790 INFO:teuthology.orchestra.run.smithi071.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-19T01:29:39.790 INFO:teuthology.orchestra.run.smithi071.stdout:Recommended packages: 2024-04-19T01:29:39.790 INFO:teuthology.orchestra.run.smithi071.stdout: thermald 2024-04-19T01:29:39.825 INFO:teuthology.orchestra.run.smithi071.stdout:The following NEW packages will be installed: 2024-04-19T01:29:39.825 INFO:teuthology.orchestra.run.smithi071.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-19T01:29:39.825 INFO:teuthology.orchestra.run.smithi071.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-19T01:29:39.825 INFO:teuthology.orchestra.run.smithi071.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-19T01:29:39.826 INFO:teuthology.orchestra.run.smithi071.stdout:The following packages will be upgraded: 2024-04-19T01:29:39.827 INFO:teuthology.orchestra.run.smithi071.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-19T01:29:39.889 INFO:teuthology.orchestra.run.smithi071.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-19T01:29:39.889 INFO:teuthology.orchestra.run.smithi071.stdout:Need to get 113 MB of archives. 2024-04-19T01:29:39.889 INFO:teuthology.orchestra.run.smithi071.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-19T01:29:39.889 INFO:teuthology.orchestra.run.smithi071.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-19T01:29:39.967 INFO:teuthology.orchestra.run.smithi062.stdout:Reading package lists... 2024-04-19T01:29:39.984 DEBUG:teuthology.orchestra.run.smithi062:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-19T01:29:40.035 INFO:teuthology.orchestra.run.smithi062.stdout:Reading package lists... 2024-04-19T01:29:40.218 INFO:teuthology.orchestra.run.smithi062.stdout:Building dependency tree... 2024-04-19T01:29:40.218 INFO:teuthology.orchestra.run.smithi062.stdout:Reading state information... 2024-04-19T01:29:40.379 INFO:teuthology.orchestra.run.smithi062.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T01:29:40.380 INFO:teuthology.orchestra.run.smithi062.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T01:29:40.380 INFO:teuthology.orchestra.run.smithi062.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T01:29:40.380 INFO:teuthology.orchestra.run.smithi062.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T01:29:40.380 INFO:teuthology.orchestra.run.smithi062.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-19T01:29:40.381 INFO:teuthology.orchestra.run.smithi062.stdout: python2.7-minimal 2024-04-19T01:29:40.381 INFO:teuthology.orchestra.run.smithi062.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T01:29:40.381 INFO:teuthology.orchestra.run.smithi062.stdout:The following additional packages will be installed: 2024-04-19T01:29:40.382 INFO:teuthology.orchestra.run.smithi062.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-19T01:29:40.382 INFO:teuthology.orchestra.run.smithi062.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-19T01:29:40.382 INFO:teuthology.orchestra.run.smithi062.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-19T01:29:40.383 INFO:teuthology.orchestra.run.smithi062.stdout:Suggested packages: 2024-04-19T01:29:40.383 INFO:teuthology.orchestra.run.smithi062.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-19T01:29:40.383 INFO:teuthology.orchestra.run.smithi062.stdout:Recommended packages: 2024-04-19T01:29:40.383 INFO:teuthology.orchestra.run.smithi062.stdout: thermald 2024-04-19T01:29:40.427 INFO:teuthology.orchestra.run.smithi062.stdout:The following NEW packages will be installed: 2024-04-19T01:29:40.427 INFO:teuthology.orchestra.run.smithi062.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-19T01:29:40.427 INFO:teuthology.orchestra.run.smithi062.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-19T01:29:40.427 INFO:teuthology.orchestra.run.smithi062.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-19T01:29:40.428 INFO:teuthology.orchestra.run.smithi062.stdout:The following packages will be upgraded: 2024-04-19T01:29:40.428 INFO:teuthology.orchestra.run.smithi062.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-19T01:29:40.496 INFO:teuthology.orchestra.run.smithi062.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-19T01:29:40.496 INFO:teuthology.orchestra.run.smithi062.stdout:Need to get 113 MB of archives. 2024-04-19T01:29:40.496 INFO:teuthology.orchestra.run.smithi062.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-19T01:29:40.496 INFO:teuthology.orchestra.run.smithi062.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-19T01:29:40.531 INFO:teuthology.orchestra.run.smithi071.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-19T01:29:40.809 INFO:teuthology.orchestra.run.smithi071.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-19T01:29:41.921 INFO:teuthology.orchestra.run.smithi062.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-19T01:29:42.227 INFO:teuthology.orchestra.run.smithi062.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-19T01:29:43.059 INFO:teuthology.orchestra.run.smithi071.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-19T01:29:43.059 INFO:teuthology.orchestra.run.smithi071.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-19T01:29:43.059 INFO:teuthology.orchestra.run.smithi071.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-19T01:29:43.394 INFO:teuthology.orchestra.run.smithi071.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-19T01:29:43.449 INFO:teuthology.orchestra.run.smithi071.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-19T01:29:43.843 INFO:teuthology.orchestra.run.smithi071.stdout:Fetched 113 MB in 4s (31.3 MB/s) 2024-04-19T01:29:44.019 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-19T01:29:44.106 INFO:teuthology.orchestra.run.smithi062.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-19T01:29:44.106 INFO:teuthology.orchestra.run.smithi062.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-19T01:29:44.106 INFO:teuthology.orchestra.run.smithi062.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-19T01:29:44.250 INFO:teuthology.orchestra.run.smithi062.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-19T01:29:44.276 INFO:teuthology.orchestra.run.smithi062.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-19T01:29:44.698 INFO:teuthology.orchestra.run.smithi062.stdout:Fetched 113 MB in 4s (29.5 MB/s) 2024-04-19T01:29:44.872 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-19T01:29:46.055 INFO:teuthology.orchestra.run.smithi071.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-19T01:29:46.059 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T01:29:46.150 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:29:46.825 INFO:teuthology.orchestra.run.smithi062.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-19T01:29:46.829 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T01:29:46.895 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:29:48.801 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-19T01:29:48.822 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T01:29:48.925 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:29:48.930 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-19T01:29:48.951 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T01:29:49.059 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:29:49.420 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-19T01:29:49.441 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T01:29:49.478 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:29:49.867 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-19T01:29:49.887 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T01:29:49.916 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:29:55.852 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T01:29:55.987 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T01:29:56.044 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T01:29:56.162 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T01:29:56.328 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T01:29:56.432 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T01:29:56.487 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T01:29:56.616 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T01:29:56.677 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-19T01:29:56.700 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-19T01:29:56.743 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-19T01:29:56.952 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-19T01:29:56.975 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-19T01:29:57.019 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-19T01:30:02.135 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-19T01:30:02.172 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T01:30:02.210 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:30:02.658 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-19T01:30:02.695 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T01:30:02.732 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:30:04.885 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T01:30:04.981 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T01:30:05.282 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-19T01:30:05.411 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:30:05.537 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-19T01:30:05.674 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:30:06.028 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T01:30:06.159 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T01:30:06.526 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-19T01:30:06.622 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:30:06.723 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-19T01:30:06.851 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:30:07.128 INFO:teuthology.orchestra.run.smithi062.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-19T01:30:07.128 INFO:teuthology.orchestra.run.smithi062.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-19T01:30:07.128 INFO:teuthology.orchestra.run.smithi062.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-19T01:30:07.128 INFO:teuthology.orchestra.run.smithi062.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-19T01:30:07.685 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:30:08.814 INFO:teuthology.orchestra.run.smithi071.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-19T01:30:08.814 INFO:teuthology.orchestra.run.smithi071.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-19T01:30:08.814 INFO:teuthology.orchestra.run.smithi071.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-19T01:30:08.814 INFO:teuthology.orchestra.run.smithi071.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-19T01:30:08.970 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:30:09.137 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:30:10.405 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:30:10.537 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-19T01:30:10.672 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-19T01:30:10.806 INFO:teuthology.orchestra.run.smithi062.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:30:10.873 INFO:teuthology.orchestra.run.smithi062.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-19T01:30:10.873 INFO:teuthology.orchestra.run.smithi062.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-19T01:30:11.806 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-19T01:30:11.932 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-19T01:30:12.058 INFO:teuthology.orchestra.run.smithi071.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T01:30:12.124 INFO:teuthology.orchestra.run.smithi071.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-19T01:30:12.124 INFO:teuthology.orchestra.run.smithi071.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-19T01:30:25.136 INFO:teuthology.orchestra.run.smithi062.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-19T01:30:25.136 INFO:teuthology.orchestra.run.smithi062.stdout:Sourcing file `/etc/default/grub' 2024-04-19T01:30:25.146 INFO:teuthology.orchestra.run.smithi062.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-19T01:30:25.167 INFO:teuthology.orchestra.run.smithi062.stdout:Generating grub configuration file ... 2024-04-19T01:30:25.414 INFO:teuthology.orchestra.run.smithi062.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-19T01:30:25.434 INFO:teuthology.orchestra.run.smithi062.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-19T01:30:25.709 INFO:teuthology.orchestra.run.smithi062.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-19T01:30:25.715 INFO:teuthology.orchestra.run.smithi062.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-19T01:30:25.791 INFO:teuthology.orchestra.run.smithi062.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-19T01:30:25.796 INFO:teuthology.orchestra.run.smithi062.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-19T01:30:25.940 INFO:teuthology.orchestra.run.smithi062.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-19T01:30:25.940 INFO:teuthology.orchestra.run.smithi062.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-19T01:30:25.940 INFO:teuthology.orchestra.run.smithi062.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-19T01:30:25.995 INFO:teuthology.orchestra.run.smithi062.stdout:done 2024-04-19T01:30:26.302 INFO:teuthology.orchestra.run.smithi071.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-19T01:30:26.302 INFO:teuthology.orchestra.run.smithi071.stdout:Sourcing file `/etc/default/grub' 2024-04-19T01:30:26.321 INFO:teuthology.orchestra.run.smithi071.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-19T01:30:26.347 INFO:teuthology.orchestra.run.smithi071.stdout:Generating grub configuration file ... 2024-04-19T01:30:26.427 DEBUG:teuthology.orchestra.run.smithi062:> dpkg -s linux-image-generic 2024-04-19T01:30:26.449 INFO:teuthology.orchestra.run.smithi062.stdout:Package: linux-image-generic 2024-04-19T01:30:26.449 INFO:teuthology.orchestra.run.smithi062.stdout:Status: install ok installed 2024-04-19T01:30:26.449 INFO:teuthology.orchestra.run.smithi062.stdout:Priority: optional 2024-04-19T01:30:26.449 INFO:teuthology.orchestra.run.smithi062.stdout:Section: kernel 2024-04-19T01:30:26.449 INFO:teuthology.orchestra.run.smithi062.stdout:Installed-Size: 21 2024-04-19T01:30:26.449 INFO:teuthology.orchestra.run.smithi062.stdout:Maintainer: Ubuntu Kernel Team 2024-04-19T01:30:26.449 INFO:teuthology.orchestra.run.smithi062.stdout:Architecture: amd64 2024-04-19T01:30:26.449 INFO:teuthology.orchestra.run.smithi062.stdout:Source: linux-meta 2024-04-19T01:30:26.449 INFO:teuthology.orchestra.run.smithi062.stdout:Version: 5.15.0.105.102 2024-04-19T01:30:26.450 INFO:teuthology.orchestra.run.smithi062.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-19T01:30:26.450 INFO:teuthology.orchestra.run.smithi062.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-19T01:30:26.450 INFO:teuthology.orchestra.run.smithi062.stdout:Recommends: thermald 2024-04-19T01:30:26.450 INFO:teuthology.orchestra.run.smithi062.stdout:Description: Generic Linux kernel image 2024-04-19T01:30:26.450 INFO:teuthology.orchestra.run.smithi062.stdout: This package will always depend on the latest generic kernel image 2024-04-19T01:30:26.450 INFO:teuthology.orchestra.run.smithi062.stdout: available. 2024-04-19T01:30:26.450 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-19T01:30:26.450 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-19T01:30:26.450 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-19T01:30:26.450 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-19T01:30:26.451 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi062.front.sepia.ceph.com, path=None, version=distro) 2024-04-19T01:30:26.451 DEBUG:teuthology.orchestra.run.smithi062:> sudo apt-get clean 2024-04-19T01:30:26.597 INFO:teuthology.orchestra.run.smithi071.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-19T01:30:26.618 INFO:teuthology.orchestra.run.smithi071.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-19T01:30:26.740 DEBUG:teuthology.orchestra.run.smithi062:> sudo apt-get update 2024-04-19T01:30:26.924 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T01:30:26.924 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-19T01:30:26.929 INFO:teuthology.orchestra.run.smithi071.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-19T01:30:26.935 INFO:teuthology.orchestra.run.smithi071.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-19T01:30:26.973 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-19T01:30:27.004 INFO:teuthology.orchestra.run.smithi071.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-19T01:30:27.009 INFO:teuthology.orchestra.run.smithi071.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-19T01:30:27.033 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-19T01:30:27.148 INFO:teuthology.orchestra.run.smithi071.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-19T01:30:27.148 INFO:teuthology.orchestra.run.smithi071.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-19T01:30:27.148 INFO:teuthology.orchestra.run.smithi071.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-19T01:30:27.190 INFO:teuthology.orchestra.run.smithi071.stdout:done 2024-04-19T01:30:27.664 DEBUG:teuthology.orchestra.run.smithi071:> dpkg -s linux-image-generic 2024-04-19T01:30:27.690 INFO:teuthology.orchestra.run.smithi071.stdout:Package: linux-image-generic 2024-04-19T01:30:27.690 INFO:teuthology.orchestra.run.smithi071.stdout:Status: install ok installed 2024-04-19T01:30:27.690 INFO:teuthology.orchestra.run.smithi071.stdout:Priority: optional 2024-04-19T01:30:27.690 INFO:teuthology.orchestra.run.smithi071.stdout:Section: kernel 2024-04-19T01:30:27.690 INFO:teuthology.orchestra.run.smithi071.stdout:Installed-Size: 21 2024-04-19T01:30:27.690 INFO:teuthology.orchestra.run.smithi071.stdout:Maintainer: Ubuntu Kernel Team 2024-04-19T01:30:27.690 INFO:teuthology.orchestra.run.smithi071.stdout:Architecture: amd64 2024-04-19T01:30:27.690 INFO:teuthology.orchestra.run.smithi071.stdout:Source: linux-meta 2024-04-19T01:30:27.690 INFO:teuthology.orchestra.run.smithi071.stdout:Version: 5.15.0.105.102 2024-04-19T01:30:27.691 INFO:teuthology.orchestra.run.smithi071.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-19T01:30:27.691 INFO:teuthology.orchestra.run.smithi071.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-19T01:30:27.691 INFO:teuthology.orchestra.run.smithi071.stdout:Recommends: thermald 2024-04-19T01:30:27.691 INFO:teuthology.orchestra.run.smithi071.stdout:Description: Generic Linux kernel image 2024-04-19T01:30:27.691 INFO:teuthology.orchestra.run.smithi071.stdout: This package will always depend on the latest generic kernel image 2024-04-19T01:30:27.691 INFO:teuthology.orchestra.run.smithi071.stdout: available. 2024-04-19T01:30:27.691 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-19T01:30:27.691 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-19T01:30:27.691 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-19T01:30:27.691 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-19T01:30:27.691 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi071.front.sepia.ceph.com, path=None, version=distro) 2024-04-19T01:30:27.692 DEBUG:teuthology.orchestra.run.smithi071:> sudo apt-get clean 2024-04-19T01:30:27.868 DEBUG:teuthology.orchestra.run.smithi071:> sudo apt-get update 2024-04-19T01:30:28.133 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-19T01:30:28.134 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T01:30:28.191 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-19T01:30:28.273 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-19T01:30:28.336 INFO:teuthology.orchestra.run.smithi062.stdout:Reading package lists... 2024-04-19T01:30:28.356 DEBUG:teuthology.orchestra.run.smithi062:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-19T01:30:28.407 INFO:teuthology.orchestra.run.smithi062.stdout:Reading package lists... 2024-04-19T01:30:28.595 INFO:teuthology.orchestra.run.smithi062.stdout:Building dependency tree... 2024-04-19T01:30:28.595 INFO:teuthology.orchestra.run.smithi062.stdout:Reading state information... 2024-04-19T01:30:28.758 INFO:teuthology.orchestra.run.smithi062.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-19T01:30:28.758 INFO:teuthology.orchestra.run.smithi062.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T01:30:28.758 INFO:teuthology.orchestra.run.smithi062.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T01:30:28.758 INFO:teuthology.orchestra.run.smithi062.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T01:30:28.758 INFO:teuthology.orchestra.run.smithi062.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T01:30:28.759 INFO:teuthology.orchestra.run.smithi062.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-19T01:30:28.759 INFO:teuthology.orchestra.run.smithi062.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-19T01:30:28.759 INFO:teuthology.orchestra.run.smithi062.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T01:30:28.798 INFO:teuthology.orchestra.run.smithi062.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-19T01:30:28.800 DEBUG:teuthology.orchestra.run.smithi062:> dpkg -s linux-image-generic 2024-04-19T01:30:28.856 INFO:teuthology.orchestra.run.smithi062.stdout:Package: linux-image-generic 2024-04-19T01:30:28.856 INFO:teuthology.orchestra.run.smithi062.stdout:Status: install ok installed 2024-04-19T01:30:28.856 INFO:teuthology.orchestra.run.smithi062.stdout:Priority: optional 2024-04-19T01:30:28.856 INFO:teuthology.orchestra.run.smithi062.stdout:Section: kernel 2024-04-19T01:30:28.856 INFO:teuthology.orchestra.run.smithi062.stdout:Installed-Size: 21 2024-04-19T01:30:28.856 INFO:teuthology.orchestra.run.smithi062.stdout:Maintainer: Ubuntu Kernel Team 2024-04-19T01:30:28.856 INFO:teuthology.orchestra.run.smithi062.stdout:Architecture: amd64 2024-04-19T01:30:28.856 INFO:teuthology.orchestra.run.smithi062.stdout:Source: linux-meta 2024-04-19T01:30:28.856 INFO:teuthology.orchestra.run.smithi062.stdout:Version: 5.15.0.105.102 2024-04-19T01:30:28.856 INFO:teuthology.orchestra.run.smithi062.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-19T01:30:28.856 INFO:teuthology.orchestra.run.smithi062.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-19T01:30:28.856 INFO:teuthology.orchestra.run.smithi062.stdout:Recommends: thermald 2024-04-19T01:30:28.856 INFO:teuthology.orchestra.run.smithi062.stdout:Description: Generic Linux kernel image 2024-04-19T01:30:28.856 INFO:teuthology.orchestra.run.smithi062.stdout: This package will always depend on the latest generic kernel image 2024-04-19T01:30:28.857 INFO:teuthology.orchestra.run.smithi062.stdout: available. 2024-04-19T01:30:28.857 DEBUG:teuthology.orchestra.run.smithi062:> mktemp 2024-04-19T01:30:28.901 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/tmp.uScAdkUOKo 2024-04-19T01:30:28.901 DEBUG:teuthology.orchestra.run.smithi062:> sudo cp /boot/grub/grub.cfg /tmp/tmp.uScAdkUOKo 2024-04-19T01:30:28.957 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0666 /tmp/tmp.uScAdkUOKo 2024-04-19T01:30:29.072 DEBUG:teuthology.orchestra.remote:smithi062:/tmp/tmp.uScAdkUOKo is 10KB 2024-04-19T01:30:29.084 DEBUG:teuthology.orchestra.run.smithi062:> rm -fr /tmp/tmp.uScAdkUOKo 2024-04-19T01:30:29.090 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-19T01:30:29.145 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-19T01:30:29.145 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-19T01:30:29.145 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-19T01:30:29.212 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-19T01:30:29.212 DEBUG:teuthology.orchestra.run.smithi062:> sudo update-grub 2024-04-19T01:30:29.497 INFO:teuthology.orchestra.run.smithi071.stdout:Reading package lists... 2024-04-19T01:30:29.514 DEBUG:teuthology.orchestra.run.smithi071:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-19T01:30:29.572 INFO:teuthology.orchestra.run.smithi071.stdout:Reading package lists... 2024-04-19T01:30:29.773 INFO:teuthology.orchestra.run.smithi071.stdout:Building dependency tree... 2024-04-19T01:30:29.775 INFO:teuthology.orchestra.run.smithi071.stdout:Reading state information... 2024-04-19T01:30:29.946 INFO:teuthology.orchestra.run.smithi071.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-19T01:30:29.947 INFO:teuthology.orchestra.run.smithi071.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T01:30:29.947 INFO:teuthology.orchestra.run.smithi071.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T01:30:29.947 INFO:teuthology.orchestra.run.smithi071.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T01:30:29.947 INFO:teuthology.orchestra.run.smithi071.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T01:30:29.947 INFO:teuthology.orchestra.run.smithi071.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-19T01:30:29.947 INFO:teuthology.orchestra.run.smithi071.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-19T01:30:29.948 INFO:teuthology.orchestra.run.smithi071.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T01:30:29.986 INFO:teuthology.orchestra.run.smithi071.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-19T01:30:29.988 DEBUG:teuthology.orchestra.run.smithi071:> dpkg -s linux-image-generic 2024-04-19T01:30:30.000 INFO:teuthology.orchestra.run.smithi071.stdout:Package: linux-image-generic 2024-04-19T01:30:30.000 INFO:teuthology.orchestra.run.smithi071.stdout:Status: install ok installed 2024-04-19T01:30:30.000 INFO:teuthology.orchestra.run.smithi071.stdout:Priority: optional 2024-04-19T01:30:30.000 INFO:teuthology.orchestra.run.smithi071.stdout:Section: kernel 2024-04-19T01:30:30.000 INFO:teuthology.orchestra.run.smithi071.stdout:Installed-Size: 21 2024-04-19T01:30:30.000 INFO:teuthology.orchestra.run.smithi071.stdout:Maintainer: Ubuntu Kernel Team 2024-04-19T01:30:30.000 INFO:teuthology.orchestra.run.smithi071.stdout:Architecture: amd64 2024-04-19T01:30:30.000 INFO:teuthology.orchestra.run.smithi071.stdout:Source: linux-meta 2024-04-19T01:30:30.000 INFO:teuthology.orchestra.run.smithi071.stdout:Version: 5.15.0.105.102 2024-04-19T01:30:30.000 INFO:teuthology.orchestra.run.smithi071.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-19T01:30:30.000 INFO:teuthology.orchestra.run.smithi071.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-19T01:30:30.000 INFO:teuthology.orchestra.run.smithi071.stdout:Recommends: thermald 2024-04-19T01:30:30.000 INFO:teuthology.orchestra.run.smithi071.stdout:Description: Generic Linux kernel image 2024-04-19T01:30:30.001 INFO:teuthology.orchestra.run.smithi071.stdout: This package will always depend on the latest generic kernel image 2024-04-19T01:30:30.001 INFO:teuthology.orchestra.run.smithi071.stdout: available. 2024-04-19T01:30:30.001 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2024-04-19T01:30:30.046 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.tNJNn8qvyB 2024-04-19T01:30:30.046 DEBUG:teuthology.orchestra.run.smithi071:> sudo cp /boot/grub/grub.cfg /tmp/tmp.tNJNn8qvyB 2024-04-19T01:30:30.104 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.tNJNn8qvyB 2024-04-19T01:30:30.240 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.tNJNn8qvyB is 10KB 2024-04-19T01:30:30.250 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.tNJNn8qvyB 2024-04-19T01:30:30.256 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-19T01:30:30.306 INFO:teuthology.orchestra.run.smithi062.stderr:Sourcing file `/etc/default/grub' 2024-04-19T01:30:30.307 INFO:teuthology.orchestra.run.smithi062.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-19T01:30:30.309 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-19T01:30:30.309 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-19T01:30:30.309 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-19T01:30:30.310 INFO:teuthology.orchestra.run.smithi062.stderr:Generating grub configuration file ... 2024-04-19T01:30:30.389 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-19T01:30:30.390 DEBUG:teuthology.orchestra.run.smithi071:> sudo update-grub 2024-04-19T01:30:30.477 INFO:teuthology.orchestra.run.smithi062.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-19T01:30:30.483 INFO:teuthology.orchestra.run.smithi062.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-19T01:30:30.693 INFO:teuthology.orchestra.run.smithi062.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-19T01:30:30.698 INFO:teuthology.orchestra.run.smithi062.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-19T01:30:30.737 INFO:teuthology.orchestra.run.smithi062.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-19T01:30:30.742 INFO:teuthology.orchestra.run.smithi062.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-19T01:30:30.859 INFO:teuthology.orchestra.run.smithi062.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-19T01:30:30.860 INFO:teuthology.orchestra.run.smithi062.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-19T01:30:30.860 INFO:teuthology.orchestra.run.smithi062.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-19T01:30:30.893 INFO:teuthology.orchestra.run.smithi062.stderr:done 2024-04-19T01:30:30.902 DEBUG:teuthology.orchestra.run.smithi062:> sudo shutdown -r now 2024-04-19T01:30:31.672 INFO:teuthology.orchestra.run.smithi071.stderr:Sourcing file `/etc/default/grub' 2024-04-19T01:30:31.673 INFO:teuthology.orchestra.run.smithi071.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-19T01:30:31.675 INFO:teuthology.orchestra.run.smithi071.stderr:Generating grub configuration file ... 2024-04-19T01:30:31.860 INFO:teuthology.orchestra.run.smithi071.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-19T01:30:31.866 INFO:teuthology.orchestra.run.smithi071.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-19T01:30:32.149 INFO:teuthology.orchestra.run.smithi071.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-19T01:30:32.154 INFO:teuthology.orchestra.run.smithi071.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-19T01:30:32.195 INFO:teuthology.orchestra.run.smithi071.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-19T01:30:32.200 INFO:teuthology.orchestra.run.smithi071.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-19T01:30:32.352 INFO:teuthology.orchestra.run.smithi071.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-19T01:30:32.353 INFO:teuthology.orchestra.run.smithi071.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-19T01:30:32.353 INFO:teuthology.orchestra.run.smithi071.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-19T01:30:32.363 INFO:teuthology.orchestra.run.smithi071.stderr:done 2024-04-19T01:30:32.373 DEBUG:teuthology.orchestra.run.smithi071:> sudo shutdown -r now 2024-04-19T01:31:00.920 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-19T01:31:00.920 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-04-19T01:31:00.921 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:31:02.377 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-19T01:31:02.377 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-04-19T01:31:02.377 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:31:19.455 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.62 2024-04-19T01:31:20.734 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-04-19T01:31:28.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-04-19T01:31:28.463 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:31:29.737 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-04-19T01:31:29.737 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:31:31.518 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.62 2024-04-19T01:31:32.798 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-04-19T01:31:43.526 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-04-19T01:31:43.527 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:31:44.800 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-04-19T01:31:44.801 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:32:01.950 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.62 2024-04-19T01:32:03.230 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-04-19T01:32:16.963 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-04-19T01:32:16.964 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:32:17.169 DEBUG:teuthology.orchestra.run.smithi062:> true 2024-04-19T01:32:17.682 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-04-19T01:32:17.682 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-105-generic"... 2024-04-19T01:32:17.682 DEBUG:teuthology.orchestra.run.smithi062:> uname -r 2024-04-19T01:32:17.729 INFO:teuthology.orchestra.run.smithi062.stdout:5.15.0-105-generic 2024-04-19T01:32:17.729 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-19T01:32:17.729 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-19T01:32:17.729 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-19T01:32:18.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-04-19T01:32:18.232 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:32:18.524 DEBUG:teuthology.orchestra.run.smithi071:> true 2024-04-19T01:32:18.730 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-19T01:32:18.730 DEBUG:teuthology.orchestra.run.smithi062:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-19T01:32:18.816 INFO:teuthology.orchestra.run.smithi062.stdout:ttyS1 2024-04-19T01:32:18.842 DEBUG:teuthology.parallel:result is None 2024-04-19T01:32:19.123 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-04-19T01:32:19.123 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-105-generic"... 2024-04-19T01:32:19.123 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2024-04-19T01:32:19.169 INFO:teuthology.orchestra.run.smithi071.stdout:5.15.0-105-generic 2024-04-19T01:32:19.169 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-19T01:32:19.169 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-19T01:32:19.169 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-19T01:32:20.171 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-19T01:32:20.171 DEBUG:teuthology.orchestra.run.smithi071:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-19T01:32:20.525 INFO:teuthology.orchestra.run.smithi071.stdout:ttyS1 2024-04-19T01:32:20.547 DEBUG:teuthology.parallel:result is None 2024-04-19T01:32:20.547 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-19T01:32:20.555 INFO:teuthology.task.internal:Creating test directory... 2024-04-19T01:32:20.555 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-19T01:32:20.558 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-19T01:32:20.564 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-19T01:32:20.570 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-19T01:32:20.581 INFO:teuthology.task.internal:Creating archive directory... 2024-04-19T01:32:20.581 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-19T01:32:20.606 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-19T01:32:20.632 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-19T01:32:20.641 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-19T01:32:20.641 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-19T01:32:20.662 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-19T01:32:20.677 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-19T01:32:20.686 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-19T01:32:20.687 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-19T01:32:20.696 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-19T01:32:20.697 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-19T01:32:20.704 INFO:teuthology.task.internal:Configuring sudo... 2024-04-19T01:32:20.704 DEBUG:teuthology.orchestra.run.smithi062:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-19T01:32:20.733 DEBUG:teuthology.orchestra.run.smithi071:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-19T01:32:20.758 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-19T01:32:20.768 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-19T01:32:20.769 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-19T01:32:20.790 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-19T01:32:20.805 DEBUG:teuthology.orchestra.run.smithi062:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-19T01:32:20.841 DEBUG:teuthology.orchestra.run.smithi062:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-19T01:32:20.889 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-19T01:32:20.889 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-19T01:32:20.950 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-19T01:32:20.956 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-19T01:32:21.005 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-19T01:32:21.005 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-19T01:32:21.073 DEBUG:teuthology.orchestra.run.smithi062:> sudo service rsyslog restart 2024-04-19T01:32:21.075 DEBUG:teuthology.orchestra.run.smithi071:> sudo service rsyslog restart 2024-04-19T01:32:21.152 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-19T01:32:21.159 INFO:teuthology.task.internal:Starting timer... 2024-04-19T01:32:21.160 INFO:teuthology.run_tasks:Running task pcp... 2024-04-19T01:32:21.169 INFO:teuthology.run_tasks:Running task selinux... 2024-04-19T01:32:21.177 DEBUG:teuthology.task.selinux:Excluding smithi062: OS 'ubuntu' does not support SELinux 2024-04-19T01:32:21.177 DEBUG:teuthology.task.selinux:Excluding smithi071: OS 'ubuntu' does not support SELinux 2024-04-19T01:32:21.177 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-19T01:32:21.177 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-19T01:32:21.177 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-19T01:32:21.178 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-19T01:32:21.186 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-19T01:32:21.203 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-19T01:32:21.204 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi062.front.sepia.ceph.com,smithi071.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-19T01:39:29.090 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi062.front.sepia.ceph.com'), Remote(name='ubuntu@smithi071.front.sepia.ceph.com')] 2024-04-19T01:39:29.091 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-04-19T01:39:29.092 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:39:29.177 DEBUG:teuthology.orchestra.run.smithi062:> true 2024-04-19T01:39:29.263 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-04-19T01:39:29.264 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-04-19T01:39:29.264 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T01:39:29.344 DEBUG:teuthology.orchestra.run.smithi071:> true 2024-04-19T01:39:29.418 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-04-19T01:39:29.418 INFO:teuthology.run_tasks:Running task clock... 2024-04-19T01:39:29.430 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-19T01:39:29.430 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-19T01:39:29.431 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-19T01:39:29.433 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-19T01:39:29.433 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-19T01:39:29.464 INFO:teuthology.orchestra.run.smithi062.stdout:19 Apr 01:39:29 ntpd[17176]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-19T01:39:29.464 INFO:teuthology.orchestra.run.smithi062.stdout:19 Apr 01:39:29 ntpd[17176]: Command line: ntpd -gq 2024-04-19T01:39:29.464 INFO:teuthology.orchestra.run.smithi062.stdout:19 Apr 01:39:29 ntpd[17176]: ---------------------------------------------------- 2024-04-19T01:39:29.464 INFO:teuthology.orchestra.run.smithi062.stdout:19 Apr 01:39:29 ntpd[17176]: ntp-4 is maintained by Network Time Foundation, 2024-04-19T01:39:29.464 INFO:teuthology.orchestra.run.smithi062.stdout:19 Apr 01:39:29 ntpd[17176]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-19T01:39:29.464 INFO:teuthology.orchestra.run.smithi062.stdout:19 Apr 01:39:29 ntpd[17176]: corporation. Support and training for ntp-4 are 2024-04-19T01:39:29.464 INFO:teuthology.orchestra.run.smithi062.stdout:19 Apr 01:39:29 ntpd[17176]: available at https://www.nwtime.org/support 2024-04-19T01:39:29.465 INFO:teuthology.orchestra.run.smithi062.stdout:19 Apr 01:39:29 ntpd[17176]: ---------------------------------------------------- 2024-04-19T01:39:29.465 INFO:teuthology.orchestra.run.smithi062.stdout:19 Apr 01:39:29 ntpd[17176]: proto: precision = 0.069 usec (-24) 2024-04-19T01:39:29.465 INFO:teuthology.orchestra.run.smithi062.stdout:19 Apr 01:39:29 ntpd[17176]: basedate set to 2022-02-04 2024-04-19T01:39:29.465 INFO:teuthology.orchestra.run.smithi062.stdout:19 Apr 01:39:29 ntpd[17176]: gps base set to 2022-02-06 (week 2196) 2024-04-19T01:39:29.465 INFO:teuthology.orchestra.run.smithi062.stdout:19 Apr 01:39:29 ntpd[17176]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-19T01:39:29.465 INFO:teuthology.orchestra.run.smithi062.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-19T01:39:29.465 INFO:teuthology.orchestra.run.smithi062.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-19T01:39:29.466 INFO:teuthology.orchestra.run.smithi062.stdout:19 Apr 01:39:29 ntpd[17176]: restrict ::: KOD does nothing without LIMITED. 2024-04-19T01:39:29.466 INFO:teuthology.orchestra.run.smithi062.stdout:19 Apr 01:39:29 ntpd[17176]: Listen and drop on 0 v6wildcard [::]:123 2024-04-19T01:39:29.466 INFO:teuthology.orchestra.run.smithi062.stdout:19 Apr 01:39:29 ntpd[17176]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-19T01:39:29.466 INFO:teuthology.orchestra.run.smithi062.stdout:19 Apr 01:39:29 ntpd[17176]: Listen normally on 2 lo 127.0.0.1:123 2024-04-19T01:39:29.467 INFO:teuthology.orchestra.run.smithi062.stdout:19 Apr 01:39:29 ntpd[17176]: Listen normally on 3 enp3s0f1 172.21.15.62:123 2024-04-19T01:39:29.467 INFO:teuthology.orchestra.run.smithi062.stdout:19 Apr 01:39:29 ntpd[17176]: Listen normally on 4 lo [::1]:123 2024-04-19T01:39:29.467 INFO:teuthology.orchestra.run.smithi062.stdout:19 Apr 01:39:29 ntpd[17176]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:febb:d1c3%5]:123 2024-04-19T01:39:29.467 INFO:teuthology.orchestra.run.smithi062.stdout:19 Apr 01:39:29 ntpd[17176]: Listening on routing socket on fd #22 for interface updates 2024-04-19T01:39:29.488 INFO:teuthology.orchestra.run.smithi071.stdout:19 Apr 01:39:29 ntpd[17177]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-19T01:39:29.488 INFO:teuthology.orchestra.run.smithi071.stdout:19 Apr 01:39:29 ntpd[17177]: Command line: ntpd -gq 2024-04-19T01:39:29.488 INFO:teuthology.orchestra.run.smithi071.stdout:19 Apr 01:39:29 ntpd[17177]: ---------------------------------------------------- 2024-04-19T01:39:29.488 INFO:teuthology.orchestra.run.smithi071.stdout:19 Apr 01:39:29 ntpd[17177]: ntp-4 is maintained by Network Time Foundation, 2024-04-19T01:39:29.488 INFO:teuthology.orchestra.run.smithi071.stdout:19 Apr 01:39:29 ntpd[17177]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-19T01:39:29.488 INFO:teuthology.orchestra.run.smithi071.stdout:19 Apr 01:39:29 ntpd[17177]: corporation. Support and training for ntp-4 are 2024-04-19T01:39:29.489 INFO:teuthology.orchestra.run.smithi071.stdout:19 Apr 01:39:29 ntpd[17177]: available at https://www.nwtime.org/support 2024-04-19T01:39:29.489 INFO:teuthology.orchestra.run.smithi071.stdout:19 Apr 01:39:29 ntpd[17177]: ---------------------------------------------------- 2024-04-19T01:39:29.489 INFO:teuthology.orchestra.run.smithi071.stdout:19 Apr 01:39:29 ntpd[17177]: proto: precision = 0.044 usec (-24) 2024-04-19T01:39:29.489 INFO:teuthology.orchestra.run.smithi071.stdout:19 Apr 01:39:29 ntpd[17177]: basedate set to 2022-02-04 2024-04-19T01:39:29.489 INFO:teuthology.orchestra.run.smithi071.stdout:19 Apr 01:39:29 ntpd[17177]: gps base set to 2022-02-06 (week 2196) 2024-04-19T01:39:29.490 INFO:teuthology.orchestra.run.smithi071.stdout:19 Apr 01:39:29 ntpd[17177]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-19T01:39:29.490 INFO:teuthology.orchestra.run.smithi071.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-19T01:39:29.490 INFO:teuthology.orchestra.run.smithi071.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-19T01:39:29.491 INFO:teuthology.orchestra.run.smithi071.stdout:19 Apr 01:39:29 ntpd[17177]: restrict ::: KOD does nothing without LIMITED. 2024-04-19T01:39:29.491 INFO:teuthology.orchestra.run.smithi071.stdout:19 Apr 01:39:29 ntpd[17177]: Listen and drop on 0 v6wildcard [::]:123 2024-04-19T01:39:29.491 INFO:teuthology.orchestra.run.smithi071.stdout:19 Apr 01:39:29 ntpd[17177]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-19T01:39:29.491 INFO:teuthology.orchestra.run.smithi071.stdout:19 Apr 01:39:29 ntpd[17177]: Listen normally on 2 lo 127.0.0.1:123 2024-04-19T01:39:29.491 INFO:teuthology.orchestra.run.smithi071.stdout:19 Apr 01:39:29 ntpd[17177]: Listen normally on 3 enp3s0f1 172.21.15.71:123 2024-04-19T01:39:29.491 INFO:teuthology.orchestra.run.smithi071.stdout:19 Apr 01:39:29 ntpd[17177]: Listen normally on 4 lo [::1]:123 2024-04-19T01:39:29.491 INFO:teuthology.orchestra.run.smithi071.stdout:19 Apr 01:39:29 ntpd[17177]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d765%5]:123 2024-04-19T01:39:29.491 INFO:teuthology.orchestra.run.smithi071.stdout:19 Apr 01:39:29 ntpd[17177]: Listening on routing socket on fd #22 for interface updates 2024-04-19T01:39:30.463 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:30 ntpd[17176]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:30.463 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:30 ntpd[17176]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:31.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:31 ntpd[17177]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:31.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:31 ntpd[17177]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:31.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:31 ntpd[17177]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:31.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:31 ntpd[17177]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:31.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:31 ntpd[17177]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:39:31.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:31 ntpd[17177]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:39:32.464 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:32 ntpd[17176]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:32.464 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:32 ntpd[17176]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:32.464 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:32 ntpd[17176]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:32.464 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:32 ntpd[17176]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:32.464 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:32 ntpd[17176]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:39:32.464 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:32 ntpd[17176]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:39:32.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:32 ntpd[17177]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:32.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:32 ntpd[17177]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:32.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:32 ntpd[17177]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:32.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:32 ntpd[17177]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:32.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:32 ntpd[17177]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:39:32.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:32 ntpd[17177]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:39:33.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:33 ntpd[17177]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:33.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:33 ntpd[17177]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:33.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:33 ntpd[17177]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:33.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:33 ntpd[17177]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:33.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:33 ntpd[17177]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:33.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:33 ntpd[17177]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:33.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:33 ntpd[17177]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:39:33.490 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:33 ntpd[17177]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:39:34.465 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:34 ntpd[17176]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:34.465 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:34 ntpd[17176]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:34.465 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:34 ntpd[17176]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:34.465 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:34 ntpd[17176]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:34.465 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:34 ntpd[17176]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:34.465 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:34 ntpd[17176]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:34.465 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:34 ntpd[17176]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:34.465 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:34 ntpd[17176]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:34.465 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:34 ntpd[17176]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:39:34.465 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:34 ntpd[17176]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:39:34.466 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:34 ntpd[17176]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:34.466 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:34 ntpd[17176]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:34.466 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:34 ntpd[17176]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:39:34.466 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:34 ntpd[17176]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:39:34.466 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:34 ntpd[17176]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:34.466 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:34 ntpd[17176]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:34.466 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:34 ntpd[17176]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:39:34.466 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:34 ntpd[17176]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:39:34.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:34 ntpd[17177]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:34.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:34 ntpd[17177]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:34.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:34 ntpd[17177]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:34.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:34 ntpd[17177]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:34.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:34 ntpd[17177]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:34.490 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:34 ntpd[17177]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:34.490 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:34 ntpd[17177]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:39:34.490 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:34 ntpd[17177]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:39:34.490 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:34 ntpd[17177]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:34.490 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:34 ntpd[17177]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:34.490 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:34 ntpd[17177]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:39:34.490 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:34 ntpd[17177]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:39:35.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:35 ntpd[17177]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:35.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:35 ntpd[17177]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:35.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:35 ntpd[17177]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:35.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:35 ntpd[17177]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:35.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:35 ntpd[17177]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:39:35.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:35 ntpd[17177]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:39:36.464 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:36 ntpd[17176]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:36.465 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:36 ntpd[17176]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:36.465 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:36 ntpd[17176]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:36.465 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:36 ntpd[17176]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:36.465 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:36 ntpd[17176]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:36.465 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:36 ntpd[17176]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:36.465 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:36 ntpd[17176]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:36.465 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:36 ntpd[17176]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:36.465 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:36 ntpd[17176]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:39:36.465 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:36 ntpd[17176]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:39:36.465 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:36 ntpd[17176]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:36.465 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:36 ntpd[17176]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:36.465 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:36 ntpd[17176]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:39:36.465 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:36 ntpd[17176]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:39:36.466 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:36 ntpd[17176]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:36.466 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:36 ntpd[17176]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:36.466 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:36 ntpd[17176]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:39:36.466 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:36 ntpd[17176]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:39:36.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:36 ntpd[17177]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:36.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:36 ntpd[17177]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:36.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:36 ntpd[17177]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:36.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:36 ntpd[17177]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:36.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:36 ntpd[17177]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:36.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:36 ntpd[17177]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:36.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:36 ntpd[17177]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:39:36.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:36 ntpd[17177]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:39:36.490 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:36 ntpd[17177]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:36.490 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:36 ntpd[17177]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:36.490 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:36 ntpd[17177]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:39:36.490 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:36 ntpd[17177]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:39:37.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:37 ntpd[17177]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:37.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:37 ntpd[17177]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:37.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:37 ntpd[17177]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:37.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:37 ntpd[17177]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:37.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:37 ntpd[17177]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:39:37.489 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:37 ntpd[17177]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:39:37.490 INFO:teuthology.orchestra.run.smithi071.stdout:19 Apr 01:39:37 ntpd[17177]: ntpd: time slew +0.000912 s 2024-04-19T01:39:37.490 INFO:teuthology.orchestra.run.smithi071.stdout:ntpd: time slew +0.000912s 2024-04-19T01:39:37.490 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:37 ntpd[17177]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-19T01:39:37.490 INFO:teuthology.orchestra.run.smithi071.stderr:19 Apr 01:39:37 ntpd[17177]: can't open /var/log/ntpstats/loopstats.20240419: Permission denied 2024-04-19T01:39:37.541 INFO:teuthology.orchestra.run.smithi071.stdout: remote refid st t when poll reach delay offset jitter 2024-04-19T01:39:37.541 INFO:teuthology.orchestra.run.smithi071.stdout:============================================================================== 2024-04-19T01:39:37.541 INFO:teuthology.orchestra.run.smithi071.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:39:37.541 INFO:teuthology.orchestra.run.smithi071.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:39:37.541 INFO:teuthology.orchestra.run.smithi071.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:39:37.541 INFO:teuthology.orchestra.run.smithi071.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:39:38.464 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:38 ntpd[17176]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:38.464 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:38 ntpd[17176]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:38.464 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:38 ntpd[17176]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:38.464 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:38 ntpd[17176]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:38.464 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:38 ntpd[17176]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:38.464 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:38 ntpd[17176]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:38.464 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:38 ntpd[17176]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T01:39:38.465 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:38 ntpd[17176]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T01:39:38.465 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:38 ntpd[17176]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T01:39:38.465 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:38 ntpd[17176]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T01:39:38.465 INFO:teuthology.orchestra.run.smithi062.stdout:19 Apr 01:39:38 ntpd[17176]: ntpd: time slew -0.001547 s 2024-04-19T01:39:38.465 INFO:teuthology.orchestra.run.smithi062.stdout:ntpd: time slew -0.001547s 2024-04-19T01:39:38.465 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:38 ntpd[17176]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-19T01:39:38.466 INFO:teuthology.orchestra.run.smithi062.stderr:19 Apr 01:39:38 ntpd[17176]: can't open /var/log/ntpstats/loopstats.20240419: Permission denied 2024-04-19T01:39:38.512 INFO:teuthology.orchestra.run.smithi062.stdout: remote refid st t when poll reach delay offset jitter 2024-04-19T01:39:38.512 INFO:teuthology.orchestra.run.smithi062.stdout:============================================================================== 2024-04-19T01:39:38.512 INFO:teuthology.orchestra.run.smithi062.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:39:38.512 INFO:teuthology.orchestra.run.smithi062.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:39:38.512 INFO:teuthology.orchestra.run.smithi062.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:39:38.512 INFO:teuthology.orchestra.run.smithi062.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:39:38.513 INFO:teuthology.run_tasks:Running task install... 2024-04-19T01:39:38.525 DEBUG:teuthology.task.install:project ceph 2024-04-19T01:39:38.525 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8b4ab94f9c30da63487fa2f241ce12f483378860'}} 2024-04-19T01:39:38.525 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '8b4ab94f9c30da63487fa2f241ce12f483378860'} 2024-04-19T01:39:38.525 INFO:teuthology.task.install:Using flavor: default 2024-04-19T01:39:38.530 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-19T01:39:38.531 INFO:teuthology.task.install:extra packages: [] 2024-04-19T01:39:38.531 DEBUG:teuthology.orchestra.run.smithi062:> sudo apt-key list | grep Ceph 2024-04-19T01:39:38.532 DEBUG:teuthology.orchestra.run.smithi071:> sudo apt-key list | grep Ceph 2024-04-19T01:39:38.594 INFO:teuthology.orchestra.run.smithi071.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-19T01:39:38.619 INFO:teuthology.orchestra.run.smithi062.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-19T01:39:38.621 INFO:teuthology.orchestra.run.smithi071.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-19T01:39:38.621 INFO:teuthology.orchestra.run.smithi071.stdout:uid [ unknown] Ceph.com (release key) 2024-04-19T01:39:38.622 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-19T01:39:38.622 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8b4ab94f9c30da63487fa2f241ce12f483378860 2024-04-19T01:39:38.651 INFO:teuthology.orchestra.run.smithi062.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-19T01:39:38.651 INFO:teuthology.orchestra.run.smithi062.stdout:uid [ unknown] Ceph.com (release key) 2024-04-19T01:39:38.652 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-19T01:39:38.652 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8b4ab94f9c30da63487fa2f241ce12f483378860 2024-04-19T01:39:38.771 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default/ 2024-04-19T01:39:38.772 INFO:teuthology.task.install.deb:Package version is 19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T01:39:38.886 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default/ 2024-04-19T01:39:38.886 INFO:teuthology.task.install.deb:Package version is 19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T01:39:38.948 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-19T01:39:38.948 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-19T01:39:38.962 DEBUG:teuthology.orchestra.run.smithi071:> sudo apt-get update 2024-04-19T01:39:39.052 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-19T01:39:39.052 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-19T01:39:39.103 DEBUG:teuthology.orchestra.run.smithi062:> sudo apt-get update 2024-04-19T01:39:39.134 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-19T01:39:39.134 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T01:39:39.150 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-19T01:39:39.171 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-19T01:39:39.181 INFO:teuthology.orchestra.run.smithi071.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy InRelease 2024-04-19T01:39:39.205 INFO:teuthology.orchestra.run.smithi071.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-19T01:39:39.227 INFO:teuthology.orchestra.run.smithi071.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-19T01:39:39.249 INFO:teuthology.orchestra.run.smithi071.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,408 B] 2024-04-19T01:39:39.275 INFO:teuthology.orchestra.run.smithi071.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-04-19T01:39:39.404 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T01:39:39.405 INFO:teuthology.orchestra.run.smithi062.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-19T01:39:39.405 INFO:teuthology.orchestra.run.smithi062.stdout:Ign:3 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy InRelease 2024-04-19T01:39:39.441 INFO:teuthology.orchestra.run.smithi062.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-19T01:39:39.477 INFO:teuthology.orchestra.run.smithi062.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-19T01:39:39.479 INFO:teuthology.orchestra.run.smithi062.stdout:Ign:6 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-19T01:39:39.485 INFO:teuthology.orchestra.run.smithi071.stdout:Fetched 29.2 kB in 0s (84.2 kB/s) 2024-04-19T01:39:39.516 INFO:teuthology.orchestra.run.smithi062.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,408 B] 2024-04-19T01:39:39.559 INFO:teuthology.orchestra.run.smithi062.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-04-19T01:39:39.840 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-19T01:39:40.001 INFO:teuthology.orchestra.run.smithi062.stdout:Fetched 258 kB in 1s (364 kB/s) 2024-04-19T01:39:40.385 INFO:teuthology.orchestra.run.smithi071.stdout:Reading package lists... 2024-04-19T01:39:40.401 DEBUG:teuthology.orchestra.run.smithi071:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3154-g8b4ab94f-1jammy cephadm=19.0.0-3154-g8b4ab94f-1jammy ceph-mds=19.0.0-3154-g8b4ab94f-1jammy ceph-mgr=19.0.0-3154-g8b4ab94f-1jammy ceph-common=19.0.0-3154-g8b4ab94f-1jammy ceph-fuse=19.0.0-3154-g8b4ab94f-1jammy ceph-test=19.0.0-3154-g8b4ab94f-1jammy ceph-volume=19.0.0-3154-g8b4ab94f-1jammy radosgw=19.0.0-3154-g8b4ab94f-1jammy python3-rados=19.0.0-3154-g8b4ab94f-1jammy python3-rgw=19.0.0-3154-g8b4ab94f-1jammy python3-cephfs=19.0.0-3154-g8b4ab94f-1jammy python3-rbd=19.0.0-3154-g8b4ab94f-1jammy libcephfs2=19.0.0-3154-g8b4ab94f-1jammy libcephfs-dev=19.0.0-3154-g8b4ab94f-1jammy librados2=19.0.0-3154-g8b4ab94f-1jammy librbd1=19.0.0-3154-g8b4ab94f-1jammy rbd-fuse=19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T01:39:40.466 INFO:teuthology.orchestra.run.smithi071.stdout:Reading package lists... 2024-04-19T01:39:40.667 INFO:teuthology.orchestra.run.smithi071.stdout:Building dependency tree... 2024-04-19T01:39:40.667 INFO:teuthology.orchestra.run.smithi071.stdout:Reading state information... 2024-04-19T01:39:40.864 INFO:teuthology.orchestra.run.smithi071.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T01:39:40.864 INFO:teuthology.orchestra.run.smithi071.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T01:39:40.864 INFO:teuthology.orchestra.run.smithi071.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T01:39:40.864 INFO:teuthology.orchestra.run.smithi071.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T01:39:40.864 INFO:teuthology.orchestra.run.smithi071.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-19T01:39:40.865 INFO:teuthology.orchestra.run.smithi071.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-19T01:39:40.865 INFO:teuthology.orchestra.run.smithi071.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T01:39:40.865 INFO:teuthology.orchestra.run.smithi071.stdout:The following additional packages will be installed: 2024-04-19T01:39:40.865 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-19T01:39:40.867 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-19T01:39:40.867 INFO:teuthology.orchestra.run.smithi071.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-19T01:39:40.867 INFO:teuthology.orchestra.run.smithi071.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-19T01:39:40.867 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-19T01:39:40.867 INFO:teuthology.orchestra.run.smithi071.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-19T01:39:40.867 INFO:teuthology.orchestra.run.smithi071.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-19T01:39:40.867 INFO:teuthology.orchestra.run.smithi071.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-19T01:39:40.867 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-19T01:39:40.867 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-19T01:39:40.867 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-19T01:39:40.867 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-19T01:39:40.867 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-19T01:39:40.867 INFO:teuthology.orchestra.run.smithi071.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-19T01:39:40.868 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-19T01:39:40.868 INFO:teuthology.orchestra.run.smithi071.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-19T01:39:40.868 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-19T01:39:40.868 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-19T01:39:40.868 INFO:teuthology.orchestra.run.smithi071.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-19T01:39:40.868 INFO:teuthology.orchestra.run.smithi071.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-19T01:39:40.868 INFO:teuthology.orchestra.run.smithi071.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-19T01:39:40.868 INFO:teuthology.orchestra.run.smithi071.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-19T01:39:40.868 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-19T01:39:40.868 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet zip 2024-04-19T01:39:40.868 INFO:teuthology.orchestra.run.smithi071.stdout:Suggested packages: 2024-04-19T01:39:40.869 INFO:teuthology.orchestra.run.smithi071.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-19T01:39:40.869 INFO:teuthology.orchestra.run.smithi071.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-19T01:39:40.869 INFO:teuthology.orchestra.run.smithi071.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-19T01:39:40.869 INFO:teuthology.orchestra.run.smithi071.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-19T01:39:40.869 INFO:teuthology.orchestra.run.smithi071.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-19T01:39:40.869 INFO:teuthology.orchestra.run.smithi071.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-19T01:39:40.869 INFO:teuthology.orchestra.run.smithi071.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-19T01:39:40.869 INFO:teuthology.orchestra.run.smithi071.stdout:Recommended packages: 2024-04-19T01:39:40.869 INFO:teuthology.orchestra.run.smithi071.stdout: btrfs-tools 2024-04-19T01:39:40.969 INFO:teuthology.orchestra.run.smithi071.stdout:The following NEW packages will be installed: 2024-04-19T01:39:40.969 INFO:teuthology.orchestra.run.smithi071.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-19T01:39:40.969 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-19T01:39:40.970 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-19T01:39:40.970 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-19T01:39:40.970 INFO:teuthology.orchestra.run.smithi071.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-19T01:39:40.970 INFO:teuthology.orchestra.run.smithi071.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-19T01:39:40.971 INFO:teuthology.orchestra.run.smithi071.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-19T01:39:40.971 INFO:teuthology.orchestra.run.smithi071.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-19T01:39:40.971 INFO:teuthology.orchestra.run.smithi071.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-19T01:39:40.971 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-19T01:39:40.971 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-19T01:39:40.971 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-19T01:39:40.971 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-19T01:39:40.971 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-19T01:39:40.972 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-19T01:39:40.972 INFO:teuthology.orchestra.run.smithi071.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-19T01:39:40.972 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-19T01:39:40.972 INFO:teuthology.orchestra.run.smithi071.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-19T01:39:40.972 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-19T01:39:40.972 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-19T01:39:40.972 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-19T01:39:40.972 INFO:teuthology.orchestra.run.smithi071.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-19T01:39:40.972 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-19T01:39:40.972 INFO:teuthology.orchestra.run.smithi071.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-19T01:39:40.972 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-19T01:39:40.972 INFO:teuthology.orchestra.run.smithi071.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-19T01:39:40.994 INFO:teuthology.orchestra.run.smithi062.stdout:Reading package lists... 2024-04-19T01:39:41.011 DEBUG:teuthology.orchestra.run.smithi062:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3154-g8b4ab94f-1jammy cephadm=19.0.0-3154-g8b4ab94f-1jammy ceph-mds=19.0.0-3154-g8b4ab94f-1jammy ceph-mgr=19.0.0-3154-g8b4ab94f-1jammy ceph-common=19.0.0-3154-g8b4ab94f-1jammy ceph-fuse=19.0.0-3154-g8b4ab94f-1jammy ceph-test=19.0.0-3154-g8b4ab94f-1jammy ceph-volume=19.0.0-3154-g8b4ab94f-1jammy radosgw=19.0.0-3154-g8b4ab94f-1jammy python3-rados=19.0.0-3154-g8b4ab94f-1jammy python3-rgw=19.0.0-3154-g8b4ab94f-1jammy python3-cephfs=19.0.0-3154-g8b4ab94f-1jammy python3-rbd=19.0.0-3154-g8b4ab94f-1jammy libcephfs2=19.0.0-3154-g8b4ab94f-1jammy libcephfs-dev=19.0.0-3154-g8b4ab94f-1jammy librados2=19.0.0-3154-g8b4ab94f-1jammy librbd1=19.0.0-3154-g8b4ab94f-1jammy rbd-fuse=19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T01:39:41.075 INFO:teuthology.orchestra.run.smithi062.stdout:Reading package lists... 2024-04-19T01:39:41.113 INFO:teuthology.orchestra.run.smithi071.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-19T01:39:41.113 INFO:teuthology.orchestra.run.smithi071.stdout:Need to get 181 MB of archives. 2024-04-19T01:39:41.113 INFO:teuthology.orchestra.run.smithi071.stdout:After this operation, 829 MB of additional disk space will be used. 2024-04-19T01:39:41.113 INFO:teuthology.orchestra.run.smithi071.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3154-g8b4ab94f-1jammy [3,555 kB] 2024-04-19T01:39:41.158 INFO:teuthology.orchestra.run.smithi071.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-19T01:39:41.276 INFO:teuthology.orchestra.run.smithi062.stdout:Building dependency tree... 2024-04-19T01:39:41.277 INFO:teuthology.orchestra.run.smithi062.stdout:Reading state information... 2024-04-19T01:39:41.324 INFO:teuthology.orchestra.run.smithi071.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3154-g8b4ab94f-1jammy [3,224 kB] 2024-04-19T01:39:41.422 INFO:teuthology.orchestra.run.smithi071.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3154-g8b4ab94f-1jammy [712 kB] 2024-04-19T01:39:41.445 INFO:teuthology.orchestra.run.smithi071.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3154-g8b4ab94f-1jammy [356 kB] 2024-04-19T01:39:41.457 INFO:teuthology.orchestra.run.smithi071.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3154-g8b4ab94f-1jammy [32.8 kB] 2024-04-19T01:39:41.458 INFO:teuthology.orchestra.run.smithi071.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3154-g8b4ab94f-1jammy [180 kB] 2024-04-19T01:39:41.463 INFO:teuthology.orchestra.run.smithi071.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3154-g8b4ab94f-1jammy [66.9 kB] 2024-04-19T01:39:41.465 INFO:teuthology.orchestra.run.smithi071.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3154-g8b4ab94f-1jammy [330 kB] 2024-04-19T01:39:41.476 INFO:teuthology.orchestra.run.smithi071.stdout:Get:10 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3154-g8b4ab94f-1jammy [6,894 kB] 2024-04-19T01:39:41.482 INFO:teuthology.orchestra.run.smithi062.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T01:39:41.482 INFO:teuthology.orchestra.run.smithi062.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T01:39:41.482 INFO:teuthology.orchestra.run.smithi062.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T01:39:41.482 INFO:teuthology.orchestra.run.smithi062.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T01:39:41.483 INFO:teuthology.orchestra.run.smithi062.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-19T01:39:41.483 INFO:teuthology.orchestra.run.smithi062.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-19T01:39:41.483 INFO:teuthology.orchestra.run.smithi062.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T01:39:41.483 INFO:teuthology.orchestra.run.smithi062.stdout:The following additional packages will be installed: 2024-04-19T01:39:41.484 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-19T01:39:41.484 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-19T01:39:41.485 INFO:teuthology.orchestra.run.smithi062.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-19T01:39:41.485 INFO:teuthology.orchestra.run.smithi062.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-19T01:39:41.485 INFO:teuthology.orchestra.run.smithi062.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-19T01:39:41.485 INFO:teuthology.orchestra.run.smithi062.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-19T01:39:41.485 INFO:teuthology.orchestra.run.smithi062.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-19T01:39:41.485 INFO:teuthology.orchestra.run.smithi062.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-19T01:39:41.485 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-19T01:39:41.486 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-19T01:39:41.486 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-19T01:39:41.486 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-19T01:39:41.486 INFO:teuthology.orchestra.run.smithi062.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-19T01:39:41.486 INFO:teuthology.orchestra.run.smithi062.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-19T01:39:41.486 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-19T01:39:41.486 INFO:teuthology.orchestra.run.smithi062.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-19T01:39:41.486 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-19T01:39:41.486 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-19T01:39:41.486 INFO:teuthology.orchestra.run.smithi062.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-19T01:39:41.486 INFO:teuthology.orchestra.run.smithi062.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-19T01:39:41.486 INFO:teuthology.orchestra.run.smithi062.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-19T01:39:41.486 INFO:teuthology.orchestra.run.smithi062.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-19T01:39:41.486 INFO:teuthology.orchestra.run.smithi062.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-19T01:39:41.487 INFO:teuthology.orchestra.run.smithi062.stdout: xmlstarlet zip 2024-04-19T01:39:41.487 INFO:teuthology.orchestra.run.smithi062.stdout:Suggested packages: 2024-04-19T01:39:41.487 INFO:teuthology.orchestra.run.smithi062.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-19T01:39:41.487 INFO:teuthology.orchestra.run.smithi062.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-19T01:39:41.487 INFO:teuthology.orchestra.run.smithi062.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-19T01:39:41.487 INFO:teuthology.orchestra.run.smithi062.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-19T01:39:41.487 INFO:teuthology.orchestra.run.smithi062.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-19T01:39:41.487 INFO:teuthology.orchestra.run.smithi062.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-19T01:39:41.487 INFO:teuthology.orchestra.run.smithi062.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-19T01:39:41.488 INFO:teuthology.orchestra.run.smithi062.stdout:Recommended packages: 2024-04-19T01:39:41.488 INFO:teuthology.orchestra.run.smithi062.stdout: btrfs-tools 2024-04-19T01:39:41.586 INFO:teuthology.orchestra.run.smithi071.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-19T01:39:41.593 INFO:teuthology.orchestra.run.smithi062.stdout:The following NEW packages will be installed: 2024-04-19T01:39:41.593 INFO:teuthology.orchestra.run.smithi062.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-19T01:39:41.593 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-19T01:39:41.593 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-19T01:39:41.594 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-19T01:39:41.594 INFO:teuthology.orchestra.run.smithi062.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-19T01:39:41.594 INFO:teuthology.orchestra.run.smithi062.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-19T01:39:41.595 INFO:teuthology.orchestra.run.smithi062.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-19T01:39:41.595 INFO:teuthology.orchestra.run.smithi062.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-19T01:39:41.595 INFO:teuthology.orchestra.run.smithi062.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-19T01:39:41.595 INFO:teuthology.orchestra.run.smithi062.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-19T01:39:41.595 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-19T01:39:41.595 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-19T01:39:41.595 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-19T01:39:41.595 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-19T01:39:41.595 INFO:teuthology.orchestra.run.smithi062.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-19T01:39:41.595 INFO:teuthology.orchestra.run.smithi062.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-19T01:39:41.595 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-19T01:39:41.595 INFO:teuthology.orchestra.run.smithi062.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-19T01:39:41.595 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-19T01:39:41.595 INFO:teuthology.orchestra.run.smithi062.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-19T01:39:41.595 INFO:teuthology.orchestra.run.smithi062.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-19T01:39:41.595 INFO:teuthology.orchestra.run.smithi062.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-19T01:39:41.596 INFO:teuthology.orchestra.run.smithi062.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-19T01:39:41.596 INFO:teuthology.orchestra.run.smithi062.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-19T01:39:41.596 INFO:teuthology.orchestra.run.smithi062.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-19T01:39:41.596 INFO:teuthology.orchestra.run.smithi062.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-19T01:39:41.598 INFO:teuthology.orchestra.run.smithi071.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-19T01:39:41.681 INFO:teuthology.orchestra.run.smithi071.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-19T01:39:41.704 INFO:teuthology.orchestra.run.smithi071.stdout:Get:14 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3154-g8b4ab94f-1jammy [112 kB] 2024-04-19T01:39:41.707 INFO:teuthology.orchestra.run.smithi071.stdout:Get:15 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3154-g8b4ab94f-1jammy [466 kB] 2024-04-19T01:39:41.721 INFO:teuthology.orchestra.run.smithi071.stdout:Get:16 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3154-g8b4ab94f-1jammy [25.8 MB] 2024-04-19T01:39:41.784 INFO:teuthology.orchestra.run.smithi062.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-19T01:39:41.784 INFO:teuthology.orchestra.run.smithi062.stdout:Need to get 181 MB of archives. 2024-04-19T01:39:41.784 INFO:teuthology.orchestra.run.smithi062.stdout:After this operation, 829 MB of additional disk space will be used. 2024-04-19T01:39:41.785 INFO:teuthology.orchestra.run.smithi062.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-19T01:39:41.911 INFO:teuthology.orchestra.run.smithi071.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-19T01:39:41.923 INFO:teuthology.orchestra.run.smithi071.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-19T01:39:41.953 INFO:teuthology.orchestra.run.smithi071.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-19T01:39:41.962 INFO:teuthology.orchestra.run.smithi071.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-19T01:39:41.963 INFO:teuthology.orchestra.run.smithi071.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-19T01:39:41.965 INFO:teuthology.orchestra.run.smithi071.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-19T01:39:41.970 INFO:teuthology.orchestra.run.smithi062.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3154-g8b4ab94f-1jammy [3,555 kB] 2024-04-19T01:39:41.983 INFO:teuthology.orchestra.run.smithi071.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-19T01:39:42.151 INFO:teuthology.orchestra.run.smithi071.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-19T01:39:42.206 INFO:teuthology.orchestra.run.smithi062.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-19T01:39:42.217 INFO:teuthology.orchestra.run.smithi062.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-19T01:39:42.308 INFO:teuthology.orchestra.run.smithi062.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-19T01:39:42.542 INFO:teuthology.orchestra.run.smithi062.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-19T01:39:42.552 INFO:teuthology.orchestra.run.smithi062.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-19T01:39:42.563 INFO:teuthology.orchestra.run.smithi071.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-19T01:39:42.583 INFO:teuthology.orchestra.run.smithi062.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-19T01:39:42.592 INFO:teuthology.orchestra.run.smithi062.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-19T01:39:42.593 INFO:teuthology.orchestra.run.smithi062.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-19T01:39:42.594 INFO:teuthology.orchestra.run.smithi062.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-19T01:39:42.597 INFO:teuthology.orchestra.run.smithi071.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-19T01:39:42.600 INFO:teuthology.orchestra.run.smithi071.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-19T01:39:42.612 INFO:teuthology.orchestra.run.smithi062.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-19T01:39:42.682 INFO:teuthology.orchestra.run.smithi071.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-19T01:39:42.685 INFO:teuthology.orchestra.run.smithi071.stdout:Get:29 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3154-g8b4ab94f-1jammy [5,095 kB] 2024-04-19T01:39:42.731 INFO:teuthology.orchestra.run.smithi071.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-19T01:39:42.753 INFO:teuthology.orchestra.run.smithi071.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-19T01:39:42.759 INFO:teuthology.orchestra.run.smithi071.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-19T01:39:42.760 INFO:teuthology.orchestra.run.smithi071.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-19T01:39:42.772 INFO:teuthology.orchestra.run.smithi071.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-19T01:39:42.773 INFO:teuthology.orchestra.run.smithi071.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-19T01:39:42.775 INFO:teuthology.orchestra.run.smithi071.stdout:Get:36 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3154-g8b4ab94f-1jammy [244 kB] 2024-04-19T01:39:42.781 INFO:teuthology.orchestra.run.smithi062.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-19T01:39:42.781 INFO:teuthology.orchestra.run.smithi071.stdout:Get:37 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3154-g8b4ab94f-1jammy [124 kB] 2024-04-19T01:39:42.788 INFO:teuthology.orchestra.run.smithi071.stdout:Get:38 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3154-g8b4ab94f-1jammy [1,526 kB] 2024-04-19T01:39:42.838 INFO:teuthology.orchestra.run.smithi071.stdout:Get:39 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3154-g8b4ab94f-1jammy [6,260 kB] 2024-04-19T01:39:42.938 INFO:teuthology.orchestra.run.smithi071.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-19T01:39:43.021 INFO:teuthology.orchestra.run.smithi071.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-19T01:39:43.050 INFO:teuthology.orchestra.run.smithi071.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-19T01:39:43.057 INFO:teuthology.orchestra.run.smithi071.stdout:Get:43 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3154-g8b4ab94f-1jammy [22.8 MB] 2024-04-19T01:39:43.066 INFO:teuthology.orchestra.run.smithi071.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-19T01:39:43.111 INFO:teuthology.orchestra.run.smithi071.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-19T01:39:43.186 INFO:teuthology.orchestra.run.smithi062.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-19T01:39:43.221 INFO:teuthology.orchestra.run.smithi062.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-19T01:39:43.223 INFO:teuthology.orchestra.run.smithi062.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-19T01:39:43.304 INFO:teuthology.orchestra.run.smithi062.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-19T01:39:43.325 INFO:teuthology.orchestra.run.smithi071.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-19T01:39:43.340 INFO:teuthology.orchestra.run.smithi071.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-19T01:39:43.347 INFO:teuthology.orchestra.run.smithi071.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-19T01:39:43.352 INFO:teuthology.orchestra.run.smithi071.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-19T01:39:43.355 INFO:teuthology.orchestra.run.smithi062.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-19T01:39:43.373 INFO:teuthology.orchestra.run.smithi071.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-19T01:39:43.377 INFO:teuthology.orchestra.run.smithi071.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-19T01:39:43.377 INFO:teuthology.orchestra.run.smithi062.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-19T01:39:43.383 INFO:teuthology.orchestra.run.smithi062.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-19T01:39:43.385 INFO:teuthology.orchestra.run.smithi062.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-19T01:39:43.397 INFO:teuthology.orchestra.run.smithi062.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-19T01:39:43.398 INFO:teuthology.orchestra.run.smithi062.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-19T01:39:43.543 INFO:teuthology.orchestra.run.smithi071.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-19T01:39:43.566 INFO:teuthology.orchestra.run.smithi062.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-19T01:39:43.650 INFO:teuthology.orchestra.run.smithi062.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-19T01:39:43.682 INFO:teuthology.orchestra.run.smithi062.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-19T01:39:43.696 INFO:teuthology.orchestra.run.smithi062.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-19T01:39:43.743 INFO:teuthology.orchestra.run.smithi062.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-19T01:39:43.861 INFO:teuthology.orchestra.run.smithi071.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-19T01:39:43.955 INFO:teuthology.orchestra.run.smithi071.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-19T01:39:43.955 INFO:teuthology.orchestra.run.smithi071.stdout:Get:55 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3154-g8b4ab94f-1jammy [14.1 kB] 2024-04-19T01:39:43.956 INFO:teuthology.orchestra.run.smithi071.stdout:Get:56 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3154-g8b4ab94f-1jammy [870 kB] 2024-04-19T01:39:43.956 INFO:teuthology.orchestra.run.smithi071.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-19T01:39:43.973 INFO:teuthology.orchestra.run.smithi071.stdout:Get:58 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3154-g8b4ab94f-1jammy [2,437 kB] 2024-04-19T01:39:44.015 INFO:teuthology.orchestra.run.smithi071.stdout:Get:59 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3154-g8b4ab94f-1jammy [791 kB] 2024-04-19T01:39:44.025 INFO:teuthology.orchestra.run.smithi071.stdout:Get:60 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3154-g8b4ab94f-1jammy [148 kB] 2024-04-19T01:39:44.027 INFO:teuthology.orchestra.run.smithi071.stdout:Get:61 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3154-g8b4ab94f-1jammy [2,297 kB] 2024-04-19T01:39:44.068 INFO:teuthology.orchestra.run.smithi071.stdout:Get:62 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3154-g8b4ab94f-1jammy [8,625 kB] 2024-04-19T01:39:44.135 INFO:teuthology.orchestra.run.smithi071.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-19T01:39:44.136 INFO:teuthology.orchestra.run.smithi071.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-19T01:39:44.142 INFO:teuthology.orchestra.run.smithi071.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-19T01:39:44.148 INFO:teuthology.orchestra.run.smithi071.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-19T01:39:44.164 INFO:teuthology.orchestra.run.smithi071.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-19T01:39:44.187 INFO:teuthology.orchestra.run.smithi071.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-19T01:39:44.193 INFO:teuthology.orchestra.run.smithi071.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-19T01:39:44.242 INFO:teuthology.orchestra.run.smithi062.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-19T01:39:44.320 INFO:teuthology.orchestra.run.smithi062.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-19T01:39:44.327 INFO:teuthology.orchestra.run.smithi062.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-19T01:39:44.333 INFO:teuthology.orchestra.run.smithi062.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-19T01:39:44.365 INFO:teuthology.orchestra.run.smithi071.stdout:Get:70 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3154-g8b4ab94f-1jammy [14.2 kB] 2024-04-19T01:39:44.365 INFO:teuthology.orchestra.run.smithi071.stdout:Get:71 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3154-g8b4ab94f-1jammy [51.6 MB] 2024-04-19T01:39:44.367 INFO:teuthology.orchestra.run.smithi071.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-19T01:39:44.375 INFO:teuthology.orchestra.run.smithi062.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-19T01:39:44.380 INFO:teuthology.orchestra.run.smithi062.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-19T01:39:44.549 INFO:teuthology.orchestra.run.smithi062.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-19T01:39:44.621 INFO:teuthology.orchestra.run.smithi062.stdout:Get:36 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3154-g8b4ab94f-1jammy [3,224 kB] 2024-04-19T01:39:44.630 INFO:teuthology.orchestra.run.smithi071.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-19T01:39:44.809 INFO:teuthology.orchestra.run.smithi071.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-19T01:39:44.823 INFO:teuthology.orchestra.run.smithi071.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-19T01:39:44.842 INFO:teuthology.orchestra.run.smithi071.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-19T01:39:44.881 INFO:teuthology.orchestra.run.smithi062.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-19T01:39:44.907 INFO:teuthology.orchestra.run.smithi071.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-19T01:39:44.912 INFO:teuthology.orchestra.run.smithi062.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-19T01:39:44.922 INFO:teuthology.orchestra.run.smithi062.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-19T01:39:45.125 INFO:teuthology.orchestra.run.smithi062.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-19T01:39:45.126 INFO:teuthology.orchestra.run.smithi062.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-19T01:39:45.133 INFO:teuthology.orchestra.run.smithi062.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-19T01:39:45.139 INFO:teuthology.orchestra.run.smithi062.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-19T01:39:45.153 INFO:teuthology.orchestra.run.smithi062.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-19T01:39:45.175 INFO:teuthology.orchestra.run.smithi062.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-19T01:39:45.181 INFO:teuthology.orchestra.run.smithi062.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-19T01:39:45.195 INFO:teuthology.orchestra.run.smithi071.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-19T01:39:45.197 INFO:teuthology.orchestra.run.smithi071.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-19T01:39:45.200 INFO:teuthology.orchestra.run.smithi071.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-19T01:39:45.200 INFO:teuthology.orchestra.run.smithi071.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-19T01:39:45.202 INFO:teuthology.orchestra.run.smithi071.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-19T01:39:45.356 INFO:teuthology.orchestra.run.smithi062.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-19T01:39:45.430 INFO:teuthology.orchestra.run.smithi071.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-19T01:39:45.619 INFO:teuthology.orchestra.run.smithi062.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-19T01:39:45.796 INFO:teuthology.orchestra.run.smithi062.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-19T01:39:45.809 INFO:teuthology.orchestra.run.smithi062.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-19T01:39:45.829 INFO:teuthology.orchestra.run.smithi062.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-19T01:39:45.861 INFO:teuthology.orchestra.run.smithi071.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-19T01:39:45.866 INFO:teuthology.orchestra.run.smithi071.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-19T01:39:45.894 INFO:teuthology.orchestra.run.smithi062.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-19T01:39:46.186 INFO:teuthology.orchestra.run.smithi062.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-19T01:39:46.188 INFO:teuthology.orchestra.run.smithi062.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-19T01:39:46.190 INFO:teuthology.orchestra.run.smithi062.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-19T01:39:46.191 INFO:teuthology.orchestra.run.smithi062.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-19T01:39:46.193 INFO:teuthology.orchestra.run.smithi062.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-19T01:39:46.207 INFO:teuthology.orchestra.run.smithi071.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-19T01:39:46.207 INFO:teuthology.orchestra.run.smithi071.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-19T01:39:46.253 INFO:teuthology.orchestra.run.smithi071.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-19T01:39:46.254 INFO:teuthology.orchestra.run.smithi071.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-19T01:39:46.255 INFO:teuthology.orchestra.run.smithi071.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-19T01:39:46.257 INFO:teuthology.orchestra.run.smithi071.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-19T01:39:46.340 INFO:teuthology.orchestra.run.smithi071.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-19T01:39:46.341 INFO:teuthology.orchestra.run.smithi071.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-19T01:39:46.415 INFO:teuthology.orchestra.run.smithi062.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-19T01:39:46.423 INFO:teuthology.orchestra.run.smithi071.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3154-g8b4ab94f-1jammy [124 kB] 2024-04-19T01:39:46.426 INFO:teuthology.orchestra.run.smithi062.stdout:Get:59 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3154-g8b4ab94f-1jammy [712 kB] 2024-04-19T01:39:46.427 INFO:teuthology.orchestra.run.smithi071.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3154-g8b4ab94f-1jammy [38.9 kB] 2024-04-19T01:39:46.427 INFO:teuthology.orchestra.run.smithi071.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3154-g8b4ab94f-1jammy [13.6 MB] 2024-04-19T01:39:46.587 INFO:teuthology.orchestra.run.smithi071.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-19T01:39:46.709 INFO:teuthology.orchestra.run.smithi062.stdout:Get:60 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3154-g8b4ab94f-1jammy [356 kB] 2024-04-19T01:39:46.786 INFO:teuthology.orchestra.run.smithi071.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3154-g8b4ab94f-1jammy [91.4 kB] 2024-04-19T01:39:46.807 INFO:teuthology.orchestra.run.smithi062.stdout:Get:61 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3154-g8b4ab94f-1jammy [32.8 kB] 2024-04-19T01:39:46.810 INFO:teuthology.orchestra.run.smithi062.stdout:Get:62 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3154-g8b4ab94f-1jammy [180 kB] 2024-04-19T01:39:46.846 INFO:teuthology.orchestra.run.smithi062.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-19T01:39:46.847 INFO:teuthology.orchestra.run.smithi062.stdout:Get:64 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3154-g8b4ab94f-1jammy [66.9 kB] 2024-04-19T01:39:46.850 INFO:teuthology.orchestra.run.smithi062.stdout:Get:65 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3154-g8b4ab94f-1jammy [330 kB] 2024-04-19T01:39:46.851 INFO:teuthology.orchestra.run.smithi062.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-19T01:39:46.909 INFO:teuthology.orchestra.run.smithi062.stdout:Get:67 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3154-g8b4ab94f-1jammy [6,894 kB] 2024-04-19T01:39:46.994 INFO:teuthology.orchestra.run.smithi071.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-19T01:39:47.041 INFO:teuthology.orchestra.run.smithi071.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-19T01:39:47.076 INFO:teuthology.orchestra.run.smithi071.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-19T01:39:47.141 INFO:teuthology.orchestra.run.smithi062.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-19T01:39:47.142 INFO:teuthology.orchestra.run.smithi062.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-19T01:39:47.144 INFO:teuthology.orchestra.run.smithi062.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-19T01:39:47.146 INFO:teuthology.orchestra.run.smithi062.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-19T01:39:47.147 INFO:teuthology.orchestra.run.smithi062.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-19T01:39:47.149 INFO:teuthology.orchestra.run.smithi062.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-19T01:39:47.150 INFO:teuthology.orchestra.run.smithi062.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-19T01:39:47.152 INFO:teuthology.orchestra.run.smithi062.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-19T01:39:47.173 INFO:teuthology.orchestra.run.smithi071.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-19T01:39:47.216 INFO:teuthology.orchestra.run.smithi071.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-19T01:39:47.224 INFO:teuthology.orchestra.run.smithi071.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-19T01:39:47.230 INFO:teuthology.orchestra.run.smithi071.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-19T01:39:47.277 INFO:teuthology.orchestra.run.smithi071.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-19T01:39:47.280 INFO:teuthology.orchestra.run.smithi071.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-19T01:39:47.301 INFO:teuthology.orchestra.run.smithi071.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-19T01:39:47.333 INFO:teuthology.orchestra.run.smithi062.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-19T01:39:47.466 INFO:teuthology.orchestra.run.smithi071.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-19T01:39:47.739 INFO:teuthology.orchestra.run.smithi071.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-19T01:39:47.748 INFO:teuthology.orchestra.run.smithi062.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-19T01:39:47.758 INFO:teuthology.orchestra.run.smithi071.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-19T01:39:47.851 INFO:teuthology.orchestra.run.smithi062.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-19T01:39:47.869 INFO:teuthology.orchestra.run.smithi062.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-19T01:39:47.912 INFO:teuthology.orchestra.run.smithi071.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-19T01:39:47.912 INFO:teuthology.orchestra.run.smithi062.stdout:Get:80 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3154-g8b4ab94f-1jammy [112 kB] 2024-04-19T01:39:47.923 INFO:teuthology.orchestra.run.smithi062.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3154-g8b4ab94f-1jammy [466 kB] 2024-04-19T01:39:47.941 INFO:teuthology.orchestra.run.smithi071.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-19T01:39:47.972 INFO:teuthology.orchestra.run.smithi062.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-19T01:39:47.976 INFO:teuthology.orchestra.run.smithi062.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3154-g8b4ab94f-1jammy [25.8 MB] 2024-04-19T01:39:48.015 INFO:teuthology.orchestra.run.smithi062.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-19T01:39:48.024 INFO:teuthology.orchestra.run.smithi062.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-19T01:39:48.029 INFO:teuthology.orchestra.run.smithi062.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-19T01:39:48.075 INFO:teuthology.orchestra.run.smithi062.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-19T01:39:48.079 INFO:teuthology.orchestra.run.smithi062.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-19T01:39:48.099 INFO:teuthology.orchestra.run.smithi071.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-19T01:39:48.101 INFO:teuthology.orchestra.run.smithi071.stdout:Get:115 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-19T01:39:48.102 INFO:teuthology.orchestra.run.smithi062.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-19T01:39:48.103 INFO:teuthology.orchestra.run.smithi071.stdout:Get:116 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-19T01:39:48.126 INFO:teuthology.orchestra.run.smithi071.stdout:Get:117 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-19T01:39:48.130 INFO:teuthology.orchestra.run.smithi071.stdout:Get:118 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-19T01:39:48.267 INFO:teuthology.orchestra.run.smithi062.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-19T01:39:48.540 INFO:teuthology.orchestra.run.smithi062.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-19T01:39:48.558 INFO:teuthology.orchestra.run.smithi062.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-19T01:39:48.715 INFO:teuthology.orchestra.run.smithi062.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-19T01:39:48.744 INFO:teuthology.orchestra.run.smithi062.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-19T01:39:48.796 INFO:teuthology.orchestra.run.smithi071.stdout:Fetched 181 MB in 7s (25.0 MB/s) 2024-04-19T01:39:48.902 INFO:teuthology.orchestra.run.smithi062.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-19T01:39:48.904 INFO:teuthology.orchestra.run.smithi062.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-19T01:39:48.907 INFO:teuthology.orchestra.run.smithi062.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-19T01:39:48.928 INFO:teuthology.orchestra.run.smithi062.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-19T01:39:48.933 INFO:teuthology.orchestra.run.smithi062.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-19T01:39:48.982 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-19T01:39:49.083 INFO:teuthology.orchestra.run.smithi071.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-19T01:39:49.089 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-19T01:39:49.180 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-19T01:39:49.526 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-19T01:39:49.552 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-19T01:39:49.592 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-19T01:39:49.953 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-19T01:39:49.980 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-19T01:39:50.020 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-19T01:39:50.373 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-19T01:39:50.399 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T01:39:50.439 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:39:50.843 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-19T01:39:50.869 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T01:39:50.910 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:39:51.213 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-19T01:39:51.228 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T01:39:51.262 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:39:51.624 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-19T01:39:51.651 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-19T01:39:51.691 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-19T01:39:52.018 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package librados2. 2024-04-19T01:39:52.045 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../007-librados2_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:39:52.085 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking librados2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:39:52.169 INFO:teuthology.orchestra.run.smithi062.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3154-g8b4ab94f-1jammy [5,095 kB] 2024-04-19T01:39:52.538 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package librbd1. 2024-04-19T01:39:52.564 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../008-librbd1_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:39:52.604 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking librbd1 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:39:53.190 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libcephfs2. 2024-04-19T01:39:53.217 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:39:53.257 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libcephfs2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:39:53.310 INFO:teuthology.orchestra.run.smithi062.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3154-g8b4ab94f-1jammy [244 kB] 2024-04-19T01:39:53.338 INFO:teuthology.orchestra.run.smithi062.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3154-g8b4ab94f-1jammy [124 kB] 2024-04-19T01:39:53.375 INFO:teuthology.orchestra.run.smithi062.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3154-g8b4ab94f-1jammy [1,526 kB] 2024-04-19T01:39:53.526 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-rados. 2024-04-19T01:39:53.541 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../010-python3-rados_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:39:53.584 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-rados (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:39:53.677 INFO:teuthology.orchestra.run.smithi062.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3154-g8b4ab94f-1jammy [6,260 kB] 2024-04-19T01:39:53.904 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-19T01:39:53.930 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:39:53.970 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-ceph-argparse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:39:54.214 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-cephfs. 2024-04-19T01:39:54.230 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:39:54.264 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-cephfs (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:39:54.550 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-19T01:39:54.577 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:39:54.617 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-ceph-common (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:39:54.746 INFO:teuthology.orchestra.run.smithi062.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3154-g8b4ab94f-1jammy [22.8 MB] 2024-04-19T01:39:54.945 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-19T01:39:54.971 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-19T01:39:55.011 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-19T01:39:55.314 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-prettytable. 2024-04-19T01:39:55.341 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-19T01:39:55.381 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-19T01:39:55.642 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-rbd. 2024-04-19T01:39:55.668 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:39:55.708 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-rbd (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:39:56.086 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-19T01:39:56.113 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-19T01:39:56.153 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-19T01:39:56.422 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-19T01:39:56.437 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-19T01:39:56.472 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-19T01:39:56.766 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-19T01:39:56.782 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-19T01:39:56.825 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-19T01:39:57.139 INFO:teuthology.orchestra.run.smithi062.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3154-g8b4ab94f-1jammy [14.1 kB] 2024-04-19T01:39:57.139 INFO:teuthology.orchestra.run.smithi062.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3154-g8b4ab94f-1jammy [870 kB] 2024-04-19T01:39:57.145 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package lua5.1. 2024-04-19T01:39:57.156 INFO:teuthology.orchestra.run.smithi062.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3154-g8b4ab94f-1jammy [2,437 kB] 2024-04-19T01:39:57.171 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-19T01:39:57.211 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-19T01:39:57.305 INFO:teuthology.orchestra.run.smithi062.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3154-g8b4ab94f-1jammy [791 kB] 2024-04-19T01:39:57.352 INFO:teuthology.orchestra.run.smithi062.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3154-g8b4ab94f-1jammy [148 kB] 2024-04-19T01:39:57.366 INFO:teuthology.orchestra.run.smithi062.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3154-g8b4ab94f-1jammy [2,297 kB] 2024-04-19T01:39:57.517 INFO:teuthology.orchestra.run.smithi062.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3154-g8b4ab94f-1jammy [8,625 kB] 2024-04-19T01:39:57.531 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package lua-any. 2024-04-19T01:39:57.557 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-19T01:39:57.597 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-19T01:39:57.834 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package zip. 2024-04-19T01:39:57.860 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-19T01:39:57.900 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking zip (3.0-12build2) ... 2024-04-19T01:39:58.075 INFO:teuthology.orchestra.run.smithi062.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3154-g8b4ab94f-1jammy [14.2 kB] 2024-04-19T01:39:58.076 INFO:teuthology.orchestra.run.smithi062.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3154-g8b4ab94f-1jammy [51.6 MB] 2024-04-19T01:39:58.361 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package unzip. 2024-04-19T01:39:58.388 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-19T01:39:58.428 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-19T01:39:58.765 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package luarocks. 2024-04-19T01:39:58.791 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-19T01:39:58.832 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-19T01:39:59.206 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package librgw2. 2024-04-19T01:39:59.232 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../025-librgw2_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:39:59.272 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking librgw2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:39:59.850 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-rgw. 2024-04-19T01:39:59.865 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:39:59.908 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-rgw (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:00.202 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-19T01:40:00.218 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-19T01:40:00.252 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-19T01:40:00.555 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libradosstriper1. 2024-04-19T01:40:00.581 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:00.622 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libradosstriper1 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:01.050 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-common. 2024-04-19T01:40:01.076 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../029-ceph-common_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:01.116 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-common (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:01.213 INFO:teuthology.orchestra.run.smithi062.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3154-g8b4ab94f-1jammy [124 kB] 2024-04-19T01:40:01.216 INFO:teuthology.orchestra.run.smithi062.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3154-g8b4ab94f-1jammy [38.9 kB] 2024-04-19T01:40:01.217 INFO:teuthology.orchestra.run.smithi062.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3154-g8b4ab94f-1jammy [13.6 MB] 2024-04-19T01:40:01.774 INFO:teuthology.orchestra.run.smithi062.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3154-g8b4ab94f-1jammy [91.4 kB] 2024-04-19T01:40:02.420 INFO:teuthology.orchestra.run.smithi062.stdout:Fetched 181 MB in 20s (8,951 kB/s) 2024-04-19T01:40:02.555 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-base. 2024-04-19T01:40:02.578 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-19T01:40:02.581 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../030-ceph-base_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:02.622 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-base (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:02.677 INFO:teuthology.orchestra.run.smithi062.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-19T01:40:02.683 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-19T01:40:02.769 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-19T01:40:03.130 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-19T01:40:03.142 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-19T01:40:03.157 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-19T01:40:03.158 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-19T01:40:03.197 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-19T01:40:03.200 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-19T01:40:03.459 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-19T01:40:03.485 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-19T01:40:03.495 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-cheroot. 2024-04-19T01:40:03.521 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-19T01:40:03.525 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-19T01:40:03.561 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-19T01:40:03.898 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-19T01:40:03.924 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-19T01:40:03.953 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-19T01:40:03.964 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-19T01:40:03.981 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T01:40:04.028 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:40:04.250 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-19T01:40:04.277 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-19T01:40:04.317 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-19T01:40:04.407 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-19T01:40:04.434 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T01:40:04.473 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:40:04.561 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-19T01:40:04.576 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-19T01:40:04.611 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-19T01:40:04.777 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-19T01:40:04.804 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T01:40:04.843 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:40:04.906 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-tempora. 2024-04-19T01:40:04.932 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-19T01:40:04.972 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-19T01:40:05.188 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-19T01:40:05.215 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-19T01:40:05.254 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-19T01:40:05.267 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-portend. 2024-04-19T01:40:05.293 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-19T01:40:05.333 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-19T01:40:05.583 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package librados2. 2024-04-19T01:40:05.610 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../007-librados2_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:05.611 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-19T01:40:05.637 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-19T01:40:05.649 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking librados2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:05.677 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-19T01:40:05.997 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-19T01:40:06.023 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-19T01:40:06.027 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package librbd1. 2024-04-19T01:40:06.043 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../008-librbd1_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:06.055 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-19T01:40:06.078 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking librbd1 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:06.409 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-natsort. 2024-04-19T01:40:06.436 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-19T01:40:06.493 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-19T01:40:06.521 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libcephfs2. 2024-04-19T01:40:06.548 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:06.588 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libcephfs2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:06.820 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-logutils. 2024-04-19T01:40:06.847 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-19T01:40:06.886 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-19T01:40:06.907 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-rados. 2024-04-19T01:40:06.923 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../010-python3-rados_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:06.965 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-rados (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:07.190 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-19T01:40:07.216 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-19T01:40:07.256 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-19T01:40:07.268 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-19T01:40:07.295 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:40:07.327 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-ceph-argparse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:07.526 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-mako. 2024-04-19T01:40:07.529 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-cephfs. 2024-04-19T01:40:07.545 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:07.552 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-19T01:40:07.587 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-cephfs (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:07.592 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-19T01:40:07.899 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-19T01:40:07.925 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:40:07.929 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-19T01:40:07.955 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-19T01:40:07.965 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-ceph-common (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:07.995 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-19T01:40:08.285 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-19T01:40:08.312 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-19T01:40:08.352 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-19T01:40:08.461 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-19T01:40:08.487 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-19T01:40:08.531 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-19T01:40:08.654 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-prettytable. 2024-04-19T01:40:08.681 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-19T01:40:08.721 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-19T01:40:08.876 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-webob. 2024-04-19T01:40:08.902 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-19T01:40:08.924 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-rbd. 2024-04-19T01:40:08.942 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:08.950 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-19T01:40:08.982 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-rbd (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:09.327 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-19T01:40:09.354 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-waitress. 2024-04-19T01:40:09.354 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-19T01:40:09.380 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-19T01:40:09.393 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-19T01:40:09.432 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-19T01:40:09.671 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-19T01:40:09.686 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-19T01:40:09.730 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-19T01:40:09.748 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-tempita. 2024-04-19T01:40:09.774 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-19T01:40:09.814 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-19T01:40:10.034 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-19T01:40:10.050 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-19T01:40:10.092 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-19T01:40:10.101 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-paste. 2024-04-19T01:40:10.127 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-19T01:40:10.167 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-19T01:40:10.412 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package lua5.1. 2024-04-19T01:40:10.439 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-19T01:40:10.479 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-19T01:40:10.545 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-19T01:40:10.572 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-19T01:40:10.620 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-19T01:40:10.790 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package lua-any. 2024-04-19T01:40:10.806 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-19T01:40:10.840 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-19T01:40:10.932 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-19T01:40:10.958 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-19T01:40:10.998 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-19T01:40:11.059 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package zip. 2024-04-19T01:40:11.086 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-19T01:40:11.126 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking zip (3.0-12build2) ... 2024-04-19T01:40:11.309 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-webtest. 2024-04-19T01:40:11.336 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-19T01:40:11.376 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-19T01:40:11.429 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package unzip. 2024-04-19T01:40:11.456 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-19T01:40:11.495 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-19T01:40:11.662 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-pecan. 2024-04-19T01:40:11.688 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-19T01:40:11.720 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-19T01:40:11.874 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package luarocks. 2024-04-19T01:40:11.897 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-19T01:40:11.940 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-19T01:40:12.058 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-certifi. 2024-04-19T01:40:12.085 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-19T01:40:12.116 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-19T01:40:12.315 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package librgw2. 2024-04-19T01:40:12.342 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../025-librgw2_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:12.381 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking librgw2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:12.436 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-idna. 2024-04-19T01:40:12.462 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-19T01:40:12.502 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-19T01:40:12.822 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-urllib3. 2024-04-19T01:40:12.848 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-19T01:40:12.888 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-19T01:40:12.933 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-rgw. 2024-04-19T01:40:12.949 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:12.991 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-rgw (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:13.225 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-requests. 2024-04-19T01:40:13.252 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-19T01:40:13.269 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-19T01:40:13.285 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-19T01:40:13.291 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-19T01:40:13.319 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-19T01:40:13.622 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libradosstriper1. 2024-04-19T01:40:13.649 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:13.689 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libradosstriper1 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:14.108 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-common. 2024-04-19T01:40:14.135 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../029-ceph-common_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:14.175 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-common (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:14.437 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-19T01:40:14.464 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-19T01:40:14.503 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-19T01:40:14.790 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-19T01:40:14.816 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:40:14.856 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:15.228 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-19T01:40:15.254 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:15.294 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:15.614 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-19T01:40:15.640 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-19T01:40:15.688 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-19T01:40:15.975 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-openssl. 2024-04-19T01:40:15.990 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-19T01:40:16.025 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-19T01:40:16.180 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-base. 2024-04-19T01:40:16.207 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../030-ceph-base_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:16.273 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-base (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:16.328 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-mgr. 2024-04-19T01:40:16.354 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:16.394 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-mgr (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:16.789 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-mon. 2024-04-19T01:40:16.815 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:16.851 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-19T01:40:16.855 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-mon (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:16.878 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-19T01:40:17.180 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-19T01:40:17.459 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-osd. 2024-04-19T01:40:17.486 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:17.487 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-cheroot. 2024-04-19T01:40:17.514 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-19T01:40:17.542 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-osd (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:17.553 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-19T01:40:17.856 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-19T01:40:17.883 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-19T01:40:17.923 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-19T01:40:18.200 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-19T01:40:18.227 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-19T01:40:18.267 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-19T01:40:18.537 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-19T01:40:18.563 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-19T01:40:18.587 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph. 2024-04-19T01:40:18.602 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../066-ceph_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:18.603 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-19T01:40:18.645 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:18.906 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-tempora. 2024-04-19T01:40:18.933 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-19T01:40:18.972 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-19T01:40:19.173 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-fuse. 2024-04-19T01:40:19.200 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:19.239 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-fuse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:19.275 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-portend. 2024-04-19T01:40:19.302 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-19T01:40:19.342 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-19T01:40:19.626 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-mds. 2024-04-19T01:40:19.644 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-19T01:40:19.652 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:19.672 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-19T01:40:19.692 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-mds (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:19.711 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-19T01:40:19.939 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-19T01:40:19.965 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-19T01:40:20.005 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-19T01:40:20.062 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package cephadm. 2024-04-19T01:40:20.088 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../069-cephadm_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:20.120 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking cephadm (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:20.400 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-natsort. 2024-04-19T01:40:20.440 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-19T01:40:20.441 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-19T01:40:20.466 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-19T01:40:20.467 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-19T01:40:20.506 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-19T01:40:20.753 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-logutils. 2024-04-19T01:40:20.770 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-19T01:40:20.800 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-19T01:40:20.803 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-19T01:40:20.827 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-19T01:40:20.858 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-19T01:40:21.048 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-19T01:40:21.075 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-19T01:40:21.114 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-19T01:40:21.409 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-mako. 2024-04-19T01:40:21.425 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-19T01:40:21.466 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-19T01:40:21.803 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-19T01:40:21.831 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-19T01:40:21.870 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-19T01:40:21.880 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-babel. 2024-04-19T01:40:21.907 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-19T01:40:21.946 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-19T01:40:22.198 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-19T01:40:22.225 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-19T01:40:22.266 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-jinja2. 2024-04-19T01:40:22.281 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-19T01:40:22.293 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-19T01:40:22.332 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-19T01:40:22.517 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-webob. 2024-04-19T01:40:22.533 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-19T01:40:22.567 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-19T01:40:22.685 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-19T01:40:22.712 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:40:22.752 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:22.887 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-waitress. 2024-04-19T01:40:22.914 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-19T01:40:22.956 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-19T01:40:23.089 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-19T01:40:23.116 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-19T01:40:23.155 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-19T01:40:23.264 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-tempita. 2024-04-19T01:40:23.292 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-19T01:40:23.331 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-19T01:40:23.449 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-routes. 2024-04-19T01:40:23.465 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-19T01:40:23.499 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-19T01:40:23.633 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-paste. 2024-04-19T01:40:23.661 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-19T01:40:23.700 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-19T01:40:23.811 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-19T01:40:23.838 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:40:23.877 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:24.011 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-19T01:40:24.028 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-19T01:40:24.070 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-19T01:40:24.339 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-19T01:40:24.366 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-19T01:40:24.405 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-19T01:40:24.702 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-webtest. 2024-04-19T01:40:24.727 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-19T01:40:24.767 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-19T01:40:25.044 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pecan. 2024-04-19T01:40:25.060 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-19T01:40:25.094 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-19T01:40:25.382 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-19T01:40:25.398 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-19T01:40:25.432 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-19T01:40:25.449 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-certifi. 2024-04-19T01:40:25.476 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-19T01:40:25.517 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-19T01:40:25.810 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-idna. 2024-04-19T01:40:25.837 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-19T01:40:25.877 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-joblib. 2024-04-19T01:40:25.893 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-19T01:40:25.893 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-19T01:40:25.930 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-19T01:40:26.188 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-urllib3. 2024-04-19T01:40:26.215 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-19T01:40:26.230 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-19T01:40:26.254 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-19T01:40:26.257 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-19T01:40:26.297 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-19T01:40:26.549 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-requests. 2024-04-19T01:40:26.576 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-19T01:40:26.610 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-sklearn. 2024-04-19T01:40:26.615 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-19T01:40:26.637 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-19T01:40:26.676 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-19T01:40:26.918 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-19T01:40:26.945 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-19T01:40:26.984 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-19T01:40:27.229 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-19T01:40:27.246 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:40:27.288 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:27.394 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-19T01:40:27.422 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:40:27.546 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:27.826 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-19T01:40:27.852 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:27.892 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:28.170 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-19T01:40:28.197 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-19T01:40:28.228 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-19T01:40:28.497 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-openssl. 2024-04-19T01:40:28.524 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-19T01:40:28.525 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-cachetools. 2024-04-19T01:40:28.553 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-19T01:40:28.564 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-19T01:40:28.592 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-19T01:40:28.837 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-19T01:40:28.864 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-19T01:40:28.901 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-mgr. 2024-04-19T01:40:28.903 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-19T01:40:28.927 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:28.967 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-mgr (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:29.224 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-19T01:40:29.251 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-19T01:40:29.290 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-19T01:40:29.345 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-mon. 2024-04-19T01:40:29.373 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:29.411 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-mon (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:29.593 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-rsa. 2024-04-19T01:40:29.622 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-19T01:40:29.660 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-19T01:40:29.965 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-osd. 2024-04-19T01:40:29.980 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-google-auth. 2024-04-19T01:40:29.992 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:30.008 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-19T01:40:30.023 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-osd (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:30.047 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-19T01:40:30.376 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-19T01:40:30.403 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-19T01:40:30.442 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-19T01:40:30.712 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-websocket. 2024-04-19T01:40:30.740 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-19T01:40:30.779 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-19T01:40:31.059 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph. 2024-04-19T01:40:31.075 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../066-ceph_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:31.099 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-19T01:40:31.117 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:31.126 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-19T01:40:31.165 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-19T01:40:31.454 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-fuse. 2024-04-19T01:40:31.481 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:31.520 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-fuse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:31.839 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-19T01:40:31.848 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-mds. 2024-04-19T01:40:31.867 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:40:31.878 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:31.913 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:31.914 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-mds (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:32.241 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-19T01:40:32.268 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-19T01:40:32.292 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package cephadm. 2024-04-19T01:40:32.316 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-19T01:40:32.319 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../069-cephadm_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:32.359 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking cephadm (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:32.628 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-19T01:40:32.655 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-19T01:40:32.670 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-19T01:40:32.694 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-19T01:40:32.698 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-19T01:40:32.736 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-19T01:40:32.963 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package jq. 2024-04-19T01:40:32.979 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-19T01:40:33.014 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-19T01:40:33.022 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-19T01:40:33.041 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-19T01:40:33.080 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-19T01:40:33.267 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package socat. 2024-04-19T01:40:33.294 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-19T01:40:33.325 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-19T01:40:33.620 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package xmlstarlet. 2024-04-19T01:40:33.636 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-19T01:40:33.678 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-19T01:40:34.166 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-test. 2024-04-19T01:40:34.182 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../097-ceph-test_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:34.215 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-test (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:34.252 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-babel. 2024-04-19T01:40:34.280 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-19T01:40:34.318 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-19T01:40:34.621 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-jinja2. 2024-04-19T01:40:34.637 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-19T01:40:34.671 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-19T01:40:35.008 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-19T01:40:35.035 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:40:35.082 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:35.402 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-19T01:40:35.430 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-19T01:40:35.485 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-19T01:40:35.788 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-routes. 2024-04-19T01:40:35.816 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-19T01:40:35.863 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-19T01:40:36.200 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-19T01:40:36.227 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:40:36.266 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:36.712 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-volume. 2024-04-19T01:40:36.740 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:40:36.779 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-volume (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:37.091 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-19T01:40:37.119 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:37.158 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libcephfs-dev (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:37.463 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-19T01:40:37.477 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-19T01:40:37.479 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-19T01:40:37.505 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-19T01:40:37.521 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-19T01:40:37.544 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-19T01:40:37.866 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-19T01:40:37.893 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-19T01:40:37.932 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-19T01:40:37.991 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-joblib. 2024-04-19T01:40:38.007 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-19T01:40:38.049 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-19T01:40:38.287 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package nvme-cli. 2024-04-19T01:40:38.314 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-19T01:40:38.353 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-19T01:40:38.394 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-19T01:40:38.421 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-19T01:40:38.460 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-19T01:40:38.759 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package pkg-config. 2024-04-19T01:40:38.780 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-sklearn. 2024-04-19T01:40:38.786 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-19T01:40:38.807 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-19T01:40:38.825 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-19T01:40:38.846 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-19T01:40:39.112 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-19T01:40:39.128 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-19T01:40:39.170 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-19T01:40:39.458 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-19T01:40:39.471 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:40:39.590 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:39.598 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-19T01:40:39.626 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-19T01:40:39.665 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-19T01:40:39.976 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-pastescript. 2024-04-19T01:40:40.004 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-19T01:40:40.043 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-19T01:40:40.338 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-pluggy. 2024-04-19T01:40:40.366 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-19T01:40:40.404 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-19T01:40:40.535 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-cachetools. 2024-04-19T01:40:40.562 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-19T01:40:40.601 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-19T01:40:40.657 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-psutil. 2024-04-19T01:40:40.685 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-19T01:40:40.724 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-19T01:40:40.888 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-19T01:40:40.923 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-19T01:40:40.954 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-19T01:40:41.019 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-py. 2024-04-19T01:40:41.046 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-19T01:40:41.077 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-19T01:40:41.266 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-19T01:40:41.293 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-19T01:40:41.324 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-19T01:40:41.397 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-pygments. 2024-04-19T01:40:41.425 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-19T01:40:41.464 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-19T01:40:41.627 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-rsa. 2024-04-19T01:40:41.655 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-19T01:40:41.693 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-19T01:40:41.943 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-19T01:40:41.963 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-google-auth. 2024-04-19T01:40:41.971 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-19T01:40:41.991 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-19T01:40:42.026 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-19T01:40:42.030 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-19T01:40:42.313 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-toml. 2024-04-19T01:40:42.329 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-19T01:40:42.349 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-19T01:40:42.362 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-19T01:40:42.377 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-19T01:40:42.416 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-19T01:40:42.649 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-pytest. 2024-04-19T01:40:42.676 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-19T01:40:42.715 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-19T01:40:42.727 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-websocket. 2024-04-19T01:40:42.755 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-19T01:40:42.969 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-19T01:40:43.044 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-simplejson. 2024-04-19T01:40:43.072 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-19T01:40:43.111 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-19T01:40:43.264 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-19T01:40:43.292 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-19T01:40:43.330 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-19T01:40:43.363 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-19T01:40:43.380 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-19T01:40:43.413 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-19T01:40:43.935 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-19T01:40:43.952 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:40:43.993 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:44.143 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package radosgw. 2024-04-19T01:40:44.170 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../116-radosgw_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:44.209 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking radosgw (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:44.322 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-19T01:40:44.349 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-19T01:40:44.388 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-19T01:40:44.691 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-19T01:40:44.719 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-19T01:40:44.757 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-19T01:40:44.968 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package jq. 2024-04-19T01:40:44.985 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-19T01:40:45.018 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-19T01:40:45.029 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package rbd-fuse. 2024-04-19T01:40:45.045 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:45.079 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking rbd-fuse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:45.312 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package socat. 2024-04-19T01:40:45.329 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-19T01:40:45.371 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-19T01:40:45.384 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-19T01:40:45.674 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package xmlstarlet. 2024-04-19T01:40:45.690 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-19T01:40:45.748 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-19T01:40:45.876 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-19T01:40:46.002 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-19T01:40:46.119 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-test. 2024-04-19T01:40:46.147 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../097-ceph-test_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:46.185 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-test (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:46.247 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-19T01:40:46.548 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-19T01:40:46.961 INFO:teuthology.orchestra.run.smithi071.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-19T01:40:47.020 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up cephadm (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:47.512 INFO:teuthology.orchestra.run.smithi071.stdout:Adding system user cephadm....done 2024-04-19T01:40:47.661 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-19T01:40:47.937 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-19T01:40:48.171 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-19T01:40:48.289 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-19T01:40:48.457 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-volume. 2024-04-19T01:40:48.485 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T01:40:48.523 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-volume (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:48.549 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-19T01:40:48.809 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-19T01:40:48.818 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-19T01:40:48.835 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:48.876 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libcephfs-dev (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:48.910 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-19T01:40:49.198 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-19T01:40:49.203 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-19T01:40:49.226 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-19T01:40:49.264 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-19T01:40:49.555 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-19T01:40:49.585 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-19T01:40:49.613 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-19T01:40:49.652 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-19T01:40:49.815 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-19T01:40:50.031 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package nvme-cli. 2024-04-19T01:40:50.066 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-19T01:40:50.100 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-19T01:40:50.105 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-19T01:40:50.352 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-19T01:40:50.469 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package pkg-config. 2024-04-19T01:40:50.497 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-19T01:40:50.535 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-19T01:40:50.611 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-19T01:40:50.788 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-19T01:40:50.804 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-19T01:40:50.846 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-19T01:40:50.870 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-19T01:40:51.105 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-ceph-argparse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:51.266 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-19T01:40:51.294 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-19T01:40:51.332 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-19T01:40:51.448 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-19T01:40:51.582 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-19T01:40:51.643 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pastescript. 2024-04-19T01:40:51.671 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-19T01:40:51.708 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-19T01:40:51.710 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-19T01:40:51.834 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-19T01:40:52.022 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pluggy. 2024-04-19T01:40:52.038 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-19T01:40:52.071 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-19T01:40:52.094 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-19T01:40:52.324 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-psutil. 2024-04-19T01:40:52.352 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-19T01:40:52.391 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-19T01:40:52.486 INFO:teuthology.orchestra.run.smithi071.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-19T01:40:52.517 INFO:teuthology.orchestra.run.smithi071.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-19T01:40:52.572 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-19T01:40:52.698 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-19T01:40:52.746 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-py. 2024-04-19T01:40:52.775 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-19T01:40:52.818 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-19T01:40:53.066 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-19T01:40:53.146 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pygments. 2024-04-19T01:40:53.163 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-19T01:40:53.192 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-19T01:40:53.204 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-19T01:40:53.460 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-19T01:40:53.667 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-19T01:40:53.692 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-19T01:40:53.720 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-19T01:40:53.733 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-19T01:40:53.994 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-toml. 2024-04-19T01:40:54.013 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up zip (3.0-12build2) ... 2024-04-19T01:40:54.022 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-19T01:40:54.061 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-19T01:40:54.139 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-19T01:40:54.363 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pytest. 2024-04-19T01:40:54.391 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-19T01:40:54.430 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-19T01:40:54.750 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-simplejson. 2024-04-19T01:40:54.779 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-19T01:40:54.816 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-19T01:40:54.824 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-19T01:40:55.084 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-19T01:40:55.085 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-19T01:40:55.102 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-19T01:40:55.144 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-19T01:40:55.210 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-19T01:40:55.336 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-19T01:40:55.612 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-19T01:40:55.815 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package radosgw. 2024-04-19T01:40:55.843 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../116-radosgw_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:55.881 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking radosgw (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:55.980 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-19T01:40:56.231 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-19T01:40:56.617 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-19T01:40:56.675 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package rbd-fuse. 2024-04-19T01:40:56.692 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T01:40:56.725 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking rbd-fuse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:56.932 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-19T01:40:57.005 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-19T01:40:57.263 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-19T01:40:57.311 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-19T01:40:57.389 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-19T01:40:57.537 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-19T01:40:57.647 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-ceph-common (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:57.668 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-19T01:40:57.956 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-idna (3.3-1) ... 2024-04-19T01:40:58.063 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-19T01:40:58.274 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-19T01:40:58.524 INFO:teuthology.orchestra.run.smithi062.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-19T01:40:58.599 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up cephadm (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:40:59.074 INFO:teuthology.orchestra.run.smithi062.stdout:Adding system user cephadm....done 2024-04-19T01:40:59.222 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-19T01:40:59.397 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-19T01:40:59.515 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-19T01:40:59.632 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:40:59.767 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-19T01:40:59.774 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-19T01:40:59.900 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-19T01:41:00.026 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-19T01:41:00.152 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-19T01:41:00.160 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-19T01:41:00.253 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-19T01:41:00.436 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-19T01:41:00.545 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-19T01:41:00.546 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-19T01:41:00.798 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-19T01:41:00.838 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-19T01:41:01.207 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-19T01:41:01.282 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:41:01.408 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-19T01:41:01.458 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-19T01:41:01.710 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-19T01:41:01.718 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-19T01:41:01.969 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-19T01:41:02.003 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-19T01:41:02.229 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-19T01:41:02.253 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-19T01:41:02.488 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-19T01:41:02.513 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-19T01:41:02.714 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-19T01:41:02.839 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-ceph-argparse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:02.974 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-19T01:41:03.098 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-19T01:41:03.100 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-19T01:41:03.224 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-19T01:41:03.350 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-19T01:41:03.376 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-19T01:41:03.476 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-19T01:41:03.494 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-19T01:41:03.744 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-19T01:41:03.762 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-19T01:41:04.046 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-19T01:41:04.163 INFO:teuthology.orchestra.run.smithi062.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-19T01:41:04.192 INFO:teuthology.orchestra.run.smithi062.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-19T01:41:04.297 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-19T01:41:04.331 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-19T01:41:04.423 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-19T01:41:04.457 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-19T01:41:04.583 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-19T01:41:04.673 INFO:teuthology.orchestra.run.smithi071.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-19T01:41:04.799 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-19T01:41:04.900 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-19T01:41:04.927 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-19T01:41:05.028 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-19T01:41:05.151 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-19T01:41:05.396 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-19T01:41:05.427 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-19T01:41:05.657 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-19T01:41:05.737 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up zip (3.0-12build2) ... 2024-04-19T01:41:05.782 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-19T01:41:05.863 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-19T01:41:06.050 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:41:06.176 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-19T01:41:06.494 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-19T01:41:06.531 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-19T01:41:06.782 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-19T01:41:06.786 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-19T01:41:06.908 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-19T01:41:07.034 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-19T01:41:07.054 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-19T01:41:07.180 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-19T01:41:07.335 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-19T01:41:07.448 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-19T01:41:07.720 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-19T01:41:07.733 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-19T01:41:07.979 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-19T01:41:08.103 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-19T01:41:08.364 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-19T01:41:08.488 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-19T01:41:08.665 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-19T01:41:08.790 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-19T01:41:09.008 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-19T01:41:09.126 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-19T01:41:09.227 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up librados2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:09.259 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-19T01:41:09.328 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:09.385 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-ceph-common (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:09.454 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-19T01:41:09.686 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-idna (3.3-1) ... 2024-04-19T01:41:09.793 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libcephfs2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:09.894 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libradosstriper1 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:09.995 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-19T01:41:10.004 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-19T01:41:10.272 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up librbd1 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:10.398 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-mgr-modules-core (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:10.524 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-fuse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:10.738 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-19T01:41:10.738 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-19T01:41:11.218 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-19T01:41:11.301 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libcephfs-dev (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:11.452 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-rados (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:11.461 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:41:11.561 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up librgw2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:11.587 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-19T01:41:11.688 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-19T01:41:11.846 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-19T01:41:12.022 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-19T01:41:12.148 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-19T01:41:12.425 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-19T01:41:12.667 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-19T01:41:12.909 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-rbd (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:12.918 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:41:13.010 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up rbd-fuse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:13.044 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-19T01:41:13.111 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-rgw (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:13.229 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-cephfs (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:13.355 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-common (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:13.362 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-19T01:41:13.630 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-19T01:41:13.898 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-19T01:41:13.898 INFO:teuthology.orchestra.run.smithi071.stdout:Adding group ceph....done 2024-04-19T01:41:14.157 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-19T01:41:14.266 INFO:teuthology.orchestra.run.smithi071.stdout:Adding system user ceph....done 2024-04-19T01:41:14.402 INFO:teuthology.orchestra.run.smithi071.stdout:Setting system user ceph properties....done 2024-04-19T01:41:14.416 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-19T01:41:14.417 INFO:teuthology.orchestra.run.smithi071.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-19T01:41:14.553 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-19T01:41:14.693 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-19T01:41:14.827 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-19T01:41:14.887 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-19T01:41:15.111 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-19T01:41:15.237 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-19T01:41:15.366 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-test (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:15.492 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up radosgw (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:15.522 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-19T01:41:15.831 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-19T01:41:15.941 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-19T01:41:15.942 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-19T01:41:16.090 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-19T01:41:16.191 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-19T01:41:16.301 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-19T01:41:16.377 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-base (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:16.380 INFO:teuthology.orchestra.run.smithi062.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-19T01:41:16.627 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-19T01:41:16.719 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-19T01:41:16.753 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-19T01:41:17.129 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-mds (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:17.154 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-19T01:41:17.351 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-19T01:41:17.351 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-19T01:41:17.431 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-19T01:41:17.557 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-19T01:41:17.816 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T01:41:17.823 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-mgr (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:17.942 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-19T01:41:18.034 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-19T01:41:18.035 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-19T01:41:18.260 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-19T01:41:18.516 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-osd (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:18.569 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-19T01:41:18.786 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-19T01:41:18.787 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-19T01:41:18.845 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-19T01:41:18.971 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-19T01:41:19.235 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:19.247 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-19T01:41:19.336 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:19.478 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-mon (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:19.540 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-19T01:41:19.693 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-19T01:41:19.693 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-19T01:41:19.885 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-19T01:41:20.171 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-mgr-cephadm (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:20.286 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-19T01:41:20.322 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:20.449 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-mgr-dashboard (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:20.563 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-19T01:41:20.600 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-volume (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:20.907 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-19T01:41:20.928 INFO:teuthology.orchestra.run.smithi071.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-19T01:41:21.008 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up librados2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:21.029 INFO:teuthology.orchestra.run.smithi071.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-19T01:41:21.109 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:21.181 INFO:teuthology.orchestra.run.smithi071.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-19T01:41:21.235 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-19T01:41:21.546 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libcephfs2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:21.655 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libradosstriper1 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:21.790 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-19T01:41:22.058 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up librbd1 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:22.159 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-mgr-modules-core (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:22.285 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-fuse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:22.334 INFO:teuthology.orchestra.run.smithi071.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-19T01:41:22.504 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-19T01:41:22.504 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-19T01:41:22.994 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libcephfs-dev (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:23.011 INFO:teuthology.orchestra.run.smithi071.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-19T01:41:23.017 DEBUG:teuthology.parallel:result is None 2024-04-19T01:41:23.112 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-rados (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:23.213 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up librgw2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:23.330 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-19T01:41:24.568 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-rbd (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:24.693 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up rbd-fuse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:24.819 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-rgw (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:24.945 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-cephfs (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:25.071 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-common (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:25.606 INFO:teuthology.orchestra.run.smithi062.stdout:Adding group ceph....done 2024-04-19T01:41:25.990 INFO:teuthology.orchestra.run.smithi062.stdout:Adding system user ceph....done 2024-04-19T01:41:26.159 INFO:teuthology.orchestra.run.smithi062.stdout:Setting system user ceph properties....done 2024-04-19T01:41:26.174 INFO:teuthology.orchestra.run.smithi062.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-19T01:41:26.318 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-19T01:41:26.673 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-19T01:41:27.173 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-test (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:27.341 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up radosgw (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:27.837 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-19T01:41:27.837 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-19T01:41:28.370 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-base (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:28.893 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-19T01:41:29.532 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-mds (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:29.781 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-19T01:41:29.781 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-19T01:41:30.283 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-mgr (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:30.526 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-19T01:41:30.526 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-19T01:41:31.026 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-osd (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:31.304 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-19T01:41:31.304 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-19T01:41:31.803 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:31.929 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:32.080 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-mon (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:32.301 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-19T01:41:32.301 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-19T01:41:32.831 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-mgr-cephadm (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:32.974 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:33.075 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-mgr-dashboard (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:33.209 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-volume (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T01:41:33.562 INFO:teuthology.orchestra.run.smithi062.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-19T01:41:33.671 INFO:teuthology.orchestra.run.smithi062.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-19T01:41:33.807 INFO:teuthology.orchestra.run.smithi062.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-19T01:41:34.590 INFO:teuthology.orchestra.run.smithi062.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-19T01:41:35.755 INFO:teuthology.orchestra.run.smithi062.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-19T01:41:35.760 DEBUG:teuthology.parallel:result is None 2024-04-19T01:41:35.761 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8b4ab94f9c30da63487fa2f241ce12f483378860 2024-04-19T01:41:35.905 DEBUG:teuthology.orchestra.run.smithi062:> dpkg-query -W -f '${Version}' ceph 2024-04-19T01:41:35.925 INFO:teuthology.orchestra.run.smithi062.stdout:19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T01:41:35.925 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T01:41:35.926 INFO:teuthology.task.install:The correct ceph version 19.0.0-3154-g8b4ab94f-1jammy is installed. 2024-04-19T01:41:35.927 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8b4ab94f9c30da63487fa2f241ce12f483378860 2024-04-19T01:41:36.065 DEBUG:teuthology.orchestra.run.smithi071:> dpkg-query -W -f '${Version}' ceph 2024-04-19T01:41:36.084 INFO:teuthology.orchestra.run.smithi071.stdout:19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T01:41:36.084 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T01:41:36.085 INFO:teuthology.task.install:The correct ceph version 19.0.0-3154-g8b4ab94f-1jammy is installed. 2024-04-19T01:41:36.086 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-19T01:41:36.087 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-19T01:41:36.087 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-19T01:41:36.102 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-19T01:41:36.102 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-19T01:41:36.145 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-19T01:41:36.145 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-19T01:41:36.146 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/daemon-helper 2024-04-19T01:41:36.163 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-19T01:41:36.218 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-19T01:41:36.218 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/daemon-helper 2024-04-19T01:41:36.232 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-19T01:41:36.287 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-19T01:41:36.287 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-19T01:41:36.287 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-19T01:41:36.301 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-19T01:41:36.354 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-19T01:41:36.354 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-19T01:41:36.367 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-19T01:41:36.423 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-19T01:41:36.423 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-19T01:41:36.423 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/stdin-killer 2024-04-19T01:41:36.437 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-19T01:41:36.494 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-19T01:41:36.494 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/stdin-killer 2024-04-19T01:41:36.507 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-19T01:41:36.563 INFO:teuthology.run_tasks:Running task ceph... 2024-04-19T01:41:36.665 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-19T01:41:36.665 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 777 /var/log/ceph 2024-04-19T01:41:36.668 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 777 /var/log/ceph 2024-04-19T01:41:36.681 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-19T01:41:36.681 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-19T01:41:36.724 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-19T01:41:36.736 INFO:tasks.ceph:Creating extra log directories... 2024-04-19T01:41:36.736 DEBUG:teuthology.orchestra.run.smithi062:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-19T01:41:36.780 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-19T01:41:36.800 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-19T01:41:36.801 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'slow request', 'overall HEALTH_'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-19T01:41:36.801 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663309', 'branch': 'wip-yuri5-testing-2024-04-17-1400', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/pool-create-delete}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7663309', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri5-testing-2024-04-17-1400'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'slow request', 'overall HEALTH_'], 'sha1': '8b4ab94f9c30da63487fa2f241ce12f483378860'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8b4ab94f9c30da63487fa2f241ce12f483378860'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri5-testing-2024-04-17-1400', 'sha1': '8b4ab94f9c30da63487fa2f241ce12f483378860'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 1937, 'sha1': '8b4ab94f9c30da63487fa2f241ce12f483378860', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri5-testing-2024-04-17-1400', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_8b4ab94f9c30da63487fa2f241ce12f483378860/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '8b4ab94f9c30da63487fa2f241ce12f483378860', 'targets': {'smithi062.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGrkx9A8skJaBcqTAomFkrqQVf4BA59s75KrIp7KfodFGwB3b4ArrQb9AWLRBdB4PVe1pxu6vFSUKjbhUjuWZ8M=', 'smithi071.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEHCgE5GNnmLfQFcqynfoGEh+GOgSEJ8VGUFw7je1VxDrEUx1PPhOwpgb11xb3ZmToGPmFF3uU0FRm3xn29h0aM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 90, 'thrash_delay': 1, 'thrash_many': True}}, {'exec': {'client.0': ['ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-18_22:49:05', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-19T01:41:36.801 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-19T01:41:36.835 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-19T01:41:36.841 DEBUG:teuthology.orchestra.run.smithi062:> sudo install -d -m0777 -- /var/run/ceph 2024-04-19T01:41:36.883 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/run/ceph 2024-04-19T01:41:36.896 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-19T01:41:36.896 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/scratch_devs of=/dev/stdout 2024-04-19T01:41:36.943 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-19T01:41:36.943 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_1 2024-04-19T01:41:36.991 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-19T01:41:36.991 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T01:41:36.991 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 882 Links: 1 2024-04-19T01:41:36.992 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T01:41:36.992 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-04-19 01:36:10.019605322 +0000 2024-04-19T01:41:36.992 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-04-19 01:36:09.755607680 +0000 2024-04-19T01:41:36.992 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-04-19 01:36:09.755607680 +0000 2024-04-19T01:41:36.992 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: - 2024-04-19T01:41:36.992 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-19T01:41:37.045 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-04-19T01:41:37.046 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-04-19T01:41:37.046 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000384618 s, 1.3 MB/s 2024-04-19T01:41:37.047 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-19T01:41:37.095 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_2 2024-04-19T01:41:37.142 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-19T01:41:37.143 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T01:41:37.143 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 893 Links: 1 2024-04-19T01:41:37.143 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T01:41:37.143 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-04-19 01:36:10.547600605 +0000 2024-04-19T01:41:37.143 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-04-19 01:36:10.255603214 +0000 2024-04-19T01:41:37.143 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-04-19 01:36:10.255603214 +0000 2024-04-19T01:41:37.143 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: - 2024-04-19T01:41:37.143 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-19T01:41:37.197 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-04-19T01:41:37.197 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-04-19T01:41:37.197 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.00043389 s, 1.2 MB/s 2024-04-19T01:41:37.198 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-19T01:41:37.248 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_3 2024-04-19T01:41:37.294 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-19T01:41:37.295 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T01:41:37.295 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 918 Links: 1 2024-04-19T01:41:37.295 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T01:41:37.295 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-04-19 01:36:10.811598246 +0000 2024-04-19T01:41:37.296 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-04-19 01:36:10.807598283 +0000 2024-04-19T01:41:37.296 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-04-19 01:36:10.807598283 +0000 2024-04-19T01:41:37.296 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: - 2024-04-19T01:41:37.296 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-19T01:41:37.349 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-04-19T01:41:37.349 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-04-19T01:41:37.349 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000367637 s, 1.4 MB/s 2024-04-19T01:41:37.350 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-19T01:41:37.400 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_4 2024-04-19T01:41:37.446 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-19T01:41:37.447 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T01:41:37.447 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 928 Links: 1 2024-04-19T01:41:37.447 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T01:41:37.447 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-04-19 01:36:11.615591066 +0000 2024-04-19T01:41:37.447 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-04-19 01:36:11.347593459 +0000 2024-04-19T01:41:37.447 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-04-19 01:36:11.347593459 +0000 2024-04-19T01:41:37.447 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: - 2024-04-19T01:41:37.447 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-19T01:41:37.501 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-04-19T01:41:37.501 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-04-19T01:41:37.501 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000322102 s, 1.6 MB/s 2024-04-19T01:41:37.502 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-19T01:41:37.551 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-19T01:41:37.552 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-19T01:41:37.552 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/scratch_devs of=/dev/stdout 2024-04-19T01:41:37.558 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-19T01:41:37.558 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_1 2024-04-19T01:41:37.605 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-19T01:41:37.605 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T01:41:37.605 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 893 Links: 1 2024-04-19T01:41:37.605 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T01:41:37.605 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-04-19 01:36:09.944369611 +0000 2024-04-19T01:41:37.605 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-04-19 01:36:09.656372181 +0000 2024-04-19T01:41:37.605 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-04-19 01:36:09.656372181 +0000 2024-04-19T01:41:37.605 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2024-04-19T01:41:37.605 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-19T01:41:37.660 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-04-19T01:41:37.660 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-04-19T01:41:37.660 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000325078 s, 1.6 MB/s 2024-04-19T01:41:37.662 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-19T01:41:37.709 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_2 2024-04-19T01:41:37.756 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-19T01:41:37.757 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T01:41:37.757 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 910 Links: 1 2024-04-19T01:41:37.757 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T01:41:37.757 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-04-19 01:36:10.440365186 +0000 2024-04-19T01:41:37.757 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-04-19 01:36:10.164367649 +0000 2024-04-19T01:41:37.757 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-04-19 01:36:10.164367649 +0000 2024-04-19T01:41:37.757 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2024-04-19T01:41:37.757 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-19T01:41:37.811 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-04-19T01:41:37.811 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-04-19T01:41:37.811 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000411076 s, 1.2 MB/s 2024-04-19T01:41:37.812 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-19T01:41:37.862 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_3 2024-04-19T01:41:37.908 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-19T01:41:37.909 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T01:41:37.909 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 925 Links: 1 2024-04-19T01:41:37.909 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T01:41:37.909 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-04-19 01:36:10.948360655 +0000 2024-04-19T01:41:37.909 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-04-19 01:36:10.672363117 +0000 2024-04-19T01:41:37.909 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-04-19 01:36:10.672363117 +0000 2024-04-19T01:41:37.909 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2024-04-19T01:41:37.909 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-19T01:41:37.966 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-04-19T01:41:37.966 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-04-19T01:41:37.967 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000468875 s, 1.1 MB/s 2024-04-19T01:41:37.968 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-19T01:41:38.017 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_4 2024-04-19T01:41:38.064 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-19T01:41:38.064 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T01:41:38.064 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 942 Links: 1 2024-04-19T01:41:38.064 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T01:41:38.064 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-04-19 01:36:11.436356302 +0000 2024-04-19T01:41:38.064 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-04-19 01:36:11.192358479 +0000 2024-04-19T01:41:38.065 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-04-19 01:36:11.192358479 +0000 2024-04-19T01:41:38.065 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2024-04-19T01:41:38.065 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-19T01:41:38.118 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-04-19T01:41:38.118 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-04-19T01:41:38.118 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000309798 s, 1.7 MB/s 2024-04-19T01:41:38.119 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-19T01:41:38.169 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-19T01:41:38.169 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi062.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi071.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-04-19T01:41:38.169 INFO:tasks.ceph:Generating config... 2024-04-19T01:41:38.171 INFO:tasks.ceph:[client] debug monc = 20 2024-04-19T01:41:38.171 INFO:tasks.ceph:[client] debug ms = 1 2024-04-19T01:41:38.171 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-04-19T01:41:38.171 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-04-19T01:41:38.171 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-04-19T01:41:38.171 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-04-19T01:41:38.171 INFO:tasks.ceph:[global] ms inject delay type = mon 2024-04-19T01:41:38.172 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-04-19T01:41:38.172 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-04-19T01:41:38.172 INFO:tasks.ceph:[global] ms type = async 2024-04-19T01:41:38.172 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-19T01:41:38.172 INFO:tasks.ceph:[mgr] debug monc = 10 2024-04-19T01:41:38.172 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-19T01:41:38.172 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-19T01:41:38.172 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-19T01:41:38.172 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-19T01:41:38.172 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-04-19T01:41:38.172 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-04-19T01:41:38.172 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-04-19T01:41:38.173 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-04-19T01:41:38.173 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-04-19T01:41:38.173 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-04-19T01:41:38.173 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-04-19T01:41:38.173 INFO:tasks.ceph:[mon] paxos min = 10 2024-04-19T01:41:38.173 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-04-19T01:41:38.173 INFO:tasks.ceph:[mon] paxos trim min = 10 2024-04-19T01:41:38.173 INFO:tasks.ceph:[osd] bdev async discard = True 2024-04-19T01:41:38.173 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-04-19T01:41:38.173 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-04-19T01:41:38.173 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-19T01:41:38.173 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-19T01:41:38.173 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-04-19T01:41:38.174 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-19T01:41:38.174 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-19T01:41:38.174 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-19T01:41:38.174 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-19T01:41:38.174 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-19T01:41:38.174 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-19T01:41:38.174 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-19T01:41:38.174 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-19T01:41:38.174 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-04-19T01:41:38.174 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-04-19T01:41:38.174 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-19T01:41:38.175 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-04-19T01:41:38.175 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-04-19T01:41:38.175 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-19T01:41:38.175 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-04-19T01:41:38.175 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-04-19T01:41:38.175 INFO:tasks.ceph:Setting up mon.a... 2024-04-19T01:41:38.175 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-19T01:41:38.210 INFO:teuthology.orchestra.run.smithi062.stdout:creating /etc/ceph/ceph.keyring 2024-04-19T01:41:38.220 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-19T01:41:38.297 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-19T01:41:38.354 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.62'), ('mon.c', '[v2:172.21.15.62:3301,v1:172.21.15.62:6790]'), ('mon.b', '172.21.15.71')] 2024-04-19T01:41:38.354 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.62,[v2:172.21.15.62:3301,v1:172.21.15.62:6790],172.21.15.71', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-04-19T01:41:38.355 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-19T01:41:38.355 DEBUG:teuthology.orchestra.run.smithi062:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-19T01:41:38.407 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.62 --addv c '[v2:172.21.15.62:3301,v1:172.21.15.62:6790]' --add b 172.21.15.71 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:41:38.493 INFO:teuthology.orchestra.run.smithi062.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:41:38.494 INFO:teuthology.orchestra.run.smithi062.stdout:monmaptool: generated fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:41:38.494 INFO:teuthology.orchestra.run.smithi062.stdout:setting min_mon_release = quincy 2024-04-19T01:41:38.494 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 0 2024-04-19T01:41:38.494 INFO:teuthology.orchestra.run.smithi062.stdout:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:41:38.494 INFO:teuthology.orchestra.run.smithi062.stdout:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:41:38.494 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:41:38.494 INFO:teuthology.orchestra.run.smithi062.stdout:min_mon_release 17 (quincy) 2024-04-19T01:41:38.494 INFO:teuthology.orchestra.run.smithi062.stdout:election_strategy: 1 2024-04-19T01:41:38.494 INFO:teuthology.orchestra.run.smithi062.stdout:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:41:38.495 INFO:teuthology.orchestra.run.smithi062.stdout:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:41:38.495 INFO:teuthology.orchestra.run.smithi062.stdout:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:41:38.495 INFO:teuthology.orchestra.run.smithi062.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-19T01:41:38.496 DEBUG:teuthology.orchestra.run.smithi062:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-19T01:41:38.542 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 6466e4b6-2520-485f-95b3-9401915fbbad... 2024-04-19T01:41:38.543 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-19T01:41:38.587 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-19T01:41:38.617 INFO:teuthology.orchestra.run.smithi062.stdout:[global] 2024-04-19T01:41:38.617 INFO:teuthology.orchestra.run.smithi062.stdout: chdir = "" 2024-04-19T01:41:38.617 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-19T01:41:38.617 INFO:teuthology.orchestra.run.smithi062.stdout: auth supported = cephx 2024-04-19T01:41:38.617 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:41:38.618 INFO:teuthology.orchestra.run.smithi062.stdout: filestore xattr use omap = true 2024-04-19T01:41:38.618 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:41:38.618 INFO:teuthology.orchestra.run.smithi062.stdout: mon clock drift allowed = 1.000 2024-04-19T01:41:38.618 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:41:38.618 INFO:teuthology.orchestra.run.smithi062.stdout: osd crush chooseleaf type = 0 2024-04-19T01:41:38.618 INFO:teuthology.orchestra.run.smithi062.stdout: auth debug = true 2024-04-19T01:41:38.618 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:41:38.618 INFO:teuthology.orchestra.run.smithi062.stdout: ms die on old message = true 2024-04-19T01:41:38.618 INFO:teuthology.orchestra.run.smithi062.stdout: ms die on bug = true 2024-04-19T01:41:38.618 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:41:38.618 INFO:teuthology.orchestra.run.smithi062.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-19T01:41:38.618 INFO:teuthology.orchestra.run.smithi062.stdout: mon pg warn max object skew = 0 2024-04-19T01:41:38.618 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:41:38.618 INFO:teuthology.orchestra.run.smithi062.stdout: # disable pg_autoscaler by default for new pools 2024-04-19T01:41:38.619 INFO:teuthology.orchestra.run.smithi062.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-19T01:41:38.619 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:41:38.619 INFO:teuthology.orchestra.run.smithi062.stdout: osd pool default size = 2 2024-04-19T01:41:38.619 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:41:38.619 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd allow primary affinity = true 2024-04-19T01:41:38.619 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd allow pg remap = true 2024-04-19T01:41:38.619 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on legacy crush tunables = false 2024-04-19T01:41:38.619 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on crush straw calc version zero = false 2024-04-19T01:41:38.619 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on no sortbitwise = false 2024-04-19T01:41:38.619 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on osd down out interval zero = false 2024-04-19T01:41:38.619 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on too few osds = false 2024-04-19T01:41:38.619 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-19T01:41:38.619 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-19T01:41:38.619 INFO:teuthology.orchestra.run.smithi062.stdout: mon_allow_pool_size_one = true 2024-04-19T01:41:38.620 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:41:38.620 INFO:teuthology.orchestra.run.smithi062.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-19T01:41:38.620 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:41:38.620 INFO:teuthology.orchestra.run.smithi062.stdout: osd default data pool replay window = 5 2024-04-19T01:41:38.620 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:41:38.620 INFO:teuthology.orchestra.run.smithi062.stdout: mon allow pool delete = true 2024-04-19T01:41:38.620 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:41:38.620 INFO:teuthology.orchestra.run.smithi062.stdout: mon cluster log file level = debug 2024-04-19T01:41:38.620 INFO:teuthology.orchestra.run.smithi062.stdout: debug asserts on shutdown = true 2024-04-19T01:41:38.620 INFO:teuthology.orchestra.run.smithi062.stdout: mon health detail to clog = false 2024-04-19T01:41:38.620 INFO:teuthology.orchestra.run.smithi062.stdout: mon host = "172.21.15.62,[v2:172.21.15.62:3301,v1:172.21.15.62:6790],172.21.15.71" 2024-04-19T01:41:38.620 INFO:teuthology.orchestra.run.smithi062.stdout: mon client directed command retry = 5 2024-04-19T01:41:38.620 INFO:teuthology.orchestra.run.smithi062.stdout: mon election default strategy = 1 2024-04-19T01:41:38.620 INFO:teuthology.orchestra.run.smithi062.stdout: ms inject delay max = 1 2024-04-19T01:41:38.620 INFO:teuthology.orchestra.run.smithi062.stdout: ms inject delay probability = 0.005 2024-04-19T01:41:38.621 INFO:teuthology.orchestra.run.smithi062.stdout: ms inject delay type = mon 2024-04-19T01:41:38.621 INFO:teuthology.orchestra.run.smithi062.stdout: ms inject internal delays = 0.002 2024-04-19T01:41:38.621 INFO:teuthology.orchestra.run.smithi062.stdout: ms inject socket failures = 2500 2024-04-19T01:41:38.621 INFO:teuthology.orchestra.run.smithi062.stdout: ms type = async 2024-04-19T01:41:38.621 INFO:teuthology.orchestra.run.smithi062.stdout: fsid = 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:41:38.621 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:41:38.621 INFO:teuthology.orchestra.run.smithi062.stdout:[osd] 2024-04-19T01:41:38.621 INFO:teuthology.orchestra.run.smithi062.stdout: osd journal size = 100 2024-04-19T01:41:38.621 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:41:38.621 INFO:teuthology.orchestra.run.smithi062.stdout: osd scrub load threshold = 5.0 2024-04-19T01:41:38.621 INFO:teuthology.orchestra.run.smithi062.stdout: osd scrub max interval = 600 2024-04-19T01:41:38.621 INFO:teuthology.orchestra.run.smithi062.stdout: osd mclock profile = high_recovery_ops 2024-04-19T01:41:38.621 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:41:38.621 INFO:teuthology.orchestra.run.smithi062.stdout: osd recover clone overlap = true 2024-04-19T01:41:38.622 INFO:teuthology.orchestra.run.smithi062.stdout: osd recovery max chunk = 1048576 2024-04-19T01:41:38.622 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:41:38.622 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug shutdown = true 2024-04-19T01:41:38.622 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug op order = true 2024-04-19T01:41:38.622 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug verify stray on activate = true 2024-04-19T01:41:38.622 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug trim objects = true 2024-04-19T01:41:38.622 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:41:38.622 INFO:teuthology.orchestra.run.smithi062.stdout: osd open classes on start = true 2024-04-19T01:41:38.622 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug pg log writeout = true 2024-04-19T01:41:38.622 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:41:38.622 INFO:teuthology.orchestra.run.smithi062.stdout: osd deep scrub update digest min age = 30 2024-04-19T01:41:38.622 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:41:38.622 INFO:teuthology.orchestra.run.smithi062.stdout: osd map max advance = 10 2024-04-19T01:41:38.622 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:41:38.622 INFO:teuthology.orchestra.run.smithi062.stdout: journal zero on create = true 2024-04-19T01:41:38.623 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:41:38.623 INFO:teuthology.orchestra.run.smithi062.stdout: filestore ondisk finisher threads = 3 2024-04-19T01:41:38.623 INFO:teuthology.orchestra.run.smithi062.stdout: filestore apply finisher threads = 3 2024-04-19T01:41:38.623 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:41:38.623 INFO:teuthology.orchestra.run.smithi062.stdout: bdev debug aio = true 2024-04-19T01:41:38.623 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug misdirected ops = true 2024-04-19T01:41:38.623 INFO:teuthology.orchestra.run.smithi062.stdout: bdev async discard = True 2024-04-19T01:41:38.623 INFO:teuthology.orchestra.run.smithi062.stdout: bdev enable discard = True 2024-04-19T01:41:38.623 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore allocator = bitmap 2024-04-19T01:41:38.623 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore block size = 96636764160 2024-04-19T01:41:38.623 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore fsck on mount = True 2024-04-19T01:41:38.623 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore zero block detection = True 2024-04-19T01:41:38.623 INFO:teuthology.orchestra.run.smithi062.stdout: debug bluefs = 20 2024-04-19T01:41:38.623 INFO:teuthology.orchestra.run.smithi062.stdout: debug bluestore = 20 2024-04-19T01:41:38.623 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-04-19T01:41:38.623 INFO:teuthology.orchestra.run.smithi062.stdout: debug osd = 20 2024-04-19T01:41:38.623 INFO:teuthology.orchestra.run.smithi062.stdout: debug rocksdb = 10 2024-04-19T01:41:38.624 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd backfillfull_ratio = 0.85 2024-04-19T01:41:38.624 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd full ratio = 0.9 2024-04-19T01:41:38.624 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd nearfull ratio = 0.8 2024-04-19T01:41:38.624 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug verify cached snaps = True 2024-04-19T01:41:38.624 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug verify missing on start = True 2024-04-19T01:41:38.624 INFO:teuthology.orchestra.run.smithi062.stdout: osd failsafe full ratio = 0.95 2024-04-19T01:41:38.624 INFO:teuthology.orchestra.run.smithi062.stdout: osd mclock override recovery settings = True 2024-04-19T01:41:38.624 INFO:teuthology.orchestra.run.smithi062.stdout: osd objectstore = bluestore 2024-04-19T01:41:38.624 INFO:teuthology.orchestra.run.smithi062.stdout: osd op queue = debug_random 2024-04-19T01:41:38.624 INFO:teuthology.orchestra.run.smithi062.stdout: osd op queue cut off = debug_random 2024-04-19T01:41:38.624 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:41:38.624 INFO:teuthology.orchestra.run.smithi062.stdout:[mgr] 2024-04-19T01:41:38.624 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-04-19T01:41:38.624 INFO:teuthology.orchestra.run.smithi062.stdout: debug mgr = 20 2024-04-19T01:41:38.624 INFO:teuthology.orchestra.run.smithi062.stdout: debug mon = 20 2024-04-19T01:41:38.624 INFO:teuthology.orchestra.run.smithi062.stdout: debug auth = 20 2024-04-19T01:41:38.625 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min pgs per osd = 4 2024-04-19T01:41:38.625 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min bytes per osd = 10 2024-04-19T01:41:38.625 INFO:teuthology.orchestra.run.smithi062.stdout: mgr/telemetry/nag = false 2024-04-19T01:41:38.625 INFO:teuthology.orchestra.run.smithi062.stdout: debug monc = 10 2024-04-19T01:41:38.625 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:41:38.625 INFO:teuthology.orchestra.run.smithi062.stdout:[mon] 2024-04-19T01:41:38.625 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-04-19T01:41:38.625 INFO:teuthology.orchestra.run.smithi062.stdout: debug mon = 20 2024-04-19T01:41:38.625 INFO:teuthology.orchestra.run.smithi062.stdout: debug paxos = 20 2024-04-19T01:41:38.625 INFO:teuthology.orchestra.run.smithi062.stdout: debug auth = 20 2024-04-19T01:41:38.625 INFO:teuthology.orchestra.run.smithi062.stdout: mon data avail warn = 5 2024-04-19T01:41:38.625 INFO:teuthology.orchestra.run.smithi062.stdout: mon mgr mkfs grace = 240 2024-04-19T01:41:38.625 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min pgs per osd = 4 2024-04-19T01:41:38.625 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd reporter subtree level = osd 2024-04-19T01:41:38.625 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd prime pg temp = true 2024-04-19T01:41:38.625 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min bytes per osd = 10 2024-04-19T01:41:38.626 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:41:38.626 INFO:teuthology.orchestra.run.smithi062.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-19T01:41:38.626 INFO:teuthology.orchestra.run.smithi062.stdout: auth mon ticket ttl = 660 # 11m 2024-04-19T01:41:38.626 INFO:teuthology.orchestra.run.smithi062.stdout: auth service ticket ttl = 240 # 4m 2024-04-19T01:41:38.626 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:41:38.626 INFO:teuthology.orchestra.run.smithi062.stdout: # don't complain about insecure global_id in the test suite 2024-04-19T01:41:38.626 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-19T01:41:38.626 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-19T01:41:38.626 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:41:38.626 INFO:teuthology.orchestra.run.smithi062.stdout: # 1m isn't quite enough 2024-04-19T01:41:38.626 INFO:teuthology.orchestra.run.smithi062.stdout: mon_down_mkfs_grace = 2m 2024-04-19T01:41:38.626 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:41:38.626 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_filestore_osds = false 2024-04-19T01:41:38.626 INFO:teuthology.orchestra.run.smithi062.stdout: mon min osdmap epochs = 25 2024-04-19T01:41:38.626 INFO:teuthology.orchestra.run.smithi062.stdout: mon osdmap full prune interval = 2 2024-04-19T01:41:38.626 INFO:teuthology.orchestra.run.smithi062.stdout: mon osdmap full prune min = 15 2024-04-19T01:41:38.627 INFO:teuthology.orchestra.run.smithi062.stdout: mon osdmap full prune txsize = 2 2024-04-19T01:41:38.627 INFO:teuthology.orchestra.run.smithi062.stdout: mon scrub inject crc mismatch = 0.01 2024-04-19T01:41:38.627 INFO:teuthology.orchestra.run.smithi062.stdout: mon scrub inject missing keys = 0.05 2024-04-19T01:41:38.627 INFO:teuthology.orchestra.run.smithi062.stdout: mon scrub interval = 300 2024-04-19T01:41:38.627 INFO:teuthology.orchestra.run.smithi062.stdout: paxos min = 10 2024-04-19T01:41:38.627 INFO:teuthology.orchestra.run.smithi062.stdout: paxos service trim min = 5 2024-04-19T01:41:38.627 INFO:teuthology.orchestra.run.smithi062.stdout: paxos trim min = 10 2024-04-19T01:41:38.627 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:41:38.627 INFO:teuthology.orchestra.run.smithi062.stdout:[client] 2024-04-19T01:41:38.627 INFO:teuthology.orchestra.run.smithi062.stdout: rgw cache enabled = true 2024-04-19T01:41:38.627 INFO:teuthology.orchestra.run.smithi062.stdout: rgw enable ops log = true 2024-04-19T01:41:38.627 INFO:teuthology.orchestra.run.smithi062.stdout: rgw enable usage log = true 2024-04-19T01:41:38.627 INFO:teuthology.orchestra.run.smithi062.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-19T01:41:38.627 INFO:teuthology.orchestra.run.smithi062.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-19T01:41:38.627 INFO:teuthology.orchestra.run.smithi062.stdout: debug monc = 20 2024-04-19T01:41:38.627 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-04-19T01:41:38.628 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.a] 2024-04-19T01:41:38.628 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.c] 2024-04-19T01:41:38.628 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.b] 2024-04-19T01:41:38.629 INFO:teuthology.orchestra.run.smithi071.stdout:[global] 2024-04-19T01:41:38.629 INFO:teuthology.orchestra.run.smithi071.stdout: chdir = "" 2024-04-19T01:41:38.629 INFO:teuthology.orchestra.run.smithi071.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-19T01:41:38.629 INFO:teuthology.orchestra.run.smithi071.stdout: auth supported = cephx 2024-04-19T01:41:38.629 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-19T01:41:38.629 INFO:teuthology.orchestra.run.smithi071.stdout: filestore xattr use omap = true 2024-04-19T01:41:38.630 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-19T01:41:38.630 INFO:teuthology.orchestra.run.smithi071.stdout: mon clock drift allowed = 1.000 2024-04-19T01:41:38.630 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-19T01:41:38.630 INFO:teuthology.orchestra.run.smithi071.stdout: osd crush chooseleaf type = 0 2024-04-19T01:41:38.630 INFO:teuthology.orchestra.run.smithi071.stdout: auth debug = true 2024-04-19T01:41:38.630 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-19T01:41:38.630 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on old message = true 2024-04-19T01:41:38.630 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on bug = true 2024-04-19T01:41:38.630 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-19T01:41:38.630 INFO:teuthology.orchestra.run.smithi071.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-19T01:41:38.630 INFO:teuthology.orchestra.run.smithi071.stdout: mon pg warn max object skew = 0 2024-04-19T01:41:38.630 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-19T01:41:38.631 INFO:teuthology.orchestra.run.smithi071.stdout: # disable pg_autoscaler by default for new pools 2024-04-19T01:41:38.631 INFO:teuthology.orchestra.run.smithi071.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-19T01:41:38.631 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-19T01:41:38.631 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default size = 2 2024-04-19T01:41:38.631 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-19T01:41:38.631 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow primary affinity = true 2024-04-19T01:41:38.631 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow pg remap = true 2024-04-19T01:41:38.631 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on legacy crush tunables = false 2024-04-19T01:41:38.631 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on crush straw calc version zero = false 2024-04-19T01:41:38.631 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on no sortbitwise = false 2024-04-19T01:41:38.631 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on osd down out interval zero = false 2024-04-19T01:41:38.631 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on too few osds = false 2024-04-19T01:41:38.631 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-19T01:41:38.631 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-19T01:41:38.632 INFO:teuthology.orchestra.run.smithi071.stdout: mon_allow_pool_size_one = true 2024-04-19T01:41:38.632 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-19T01:41:38.632 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-19T01:41:38.632 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-19T01:41:38.632 INFO:teuthology.orchestra.run.smithi071.stdout: osd default data pool replay window = 5 2024-04-19T01:41:38.632 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-19T01:41:38.632 INFO:teuthology.orchestra.run.smithi071.stdout: mon allow pool delete = true 2024-04-19T01:41:38.632 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-19T01:41:38.632 INFO:teuthology.orchestra.run.smithi071.stdout: mon cluster log file level = debug 2024-04-19T01:41:38.632 INFO:teuthology.orchestra.run.smithi071.stdout: debug asserts on shutdown = true 2024-04-19T01:41:38.632 INFO:teuthology.orchestra.run.smithi071.stdout: mon health detail to clog = false 2024-04-19T01:41:38.632 INFO:teuthology.orchestra.run.smithi071.stdout: mon host = "172.21.15.62,[v2:172.21.15.62:3301,v1:172.21.15.62:6790],172.21.15.71" 2024-04-19T01:41:38.632 INFO:teuthology.orchestra.run.smithi071.stdout: mon client directed command retry = 5 2024-04-19T01:41:38.632 INFO:teuthology.orchestra.run.smithi071.stdout: mon election default strategy = 1 2024-04-19T01:41:38.632 INFO:teuthology.orchestra.run.smithi071.stdout: ms inject delay max = 1 2024-04-19T01:41:38.633 INFO:teuthology.orchestra.run.smithi071.stdout: ms inject delay probability = 0.005 2024-04-19T01:41:38.633 INFO:teuthology.orchestra.run.smithi071.stdout: ms inject delay type = mon 2024-04-19T01:41:38.633 INFO:teuthology.orchestra.run.smithi071.stdout: ms inject internal delays = 0.002 2024-04-19T01:41:38.633 INFO:teuthology.orchestra.run.smithi071.stdout: ms inject socket failures = 2500 2024-04-19T01:41:38.633 INFO:teuthology.orchestra.run.smithi071.stdout: ms type = async 2024-04-19T01:41:38.633 INFO:teuthology.orchestra.run.smithi071.stdout: fsid = 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:41:38.633 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-19T01:41:38.633 INFO:teuthology.orchestra.run.smithi071.stdout:[osd] 2024-04-19T01:41:38.633 INFO:teuthology.orchestra.run.smithi071.stdout: osd journal size = 100 2024-04-19T01:41:38.633 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-19T01:41:38.633 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub load threshold = 5.0 2024-04-19T01:41:38.633 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub max interval = 600 2024-04-19T01:41:38.633 INFO:teuthology.orchestra.run.smithi071.stdout: osd mclock profile = high_recovery_ops 2024-04-19T01:41:38.633 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-19T01:41:38.633 INFO:teuthology.orchestra.run.smithi071.stdout: osd recover clone overlap = true 2024-04-19T01:41:38.634 INFO:teuthology.orchestra.run.smithi071.stdout: osd recovery max chunk = 1048576 2024-04-19T01:41:38.634 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-19T01:41:38.634 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug shutdown = true 2024-04-19T01:41:38.634 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug op order = true 2024-04-19T01:41:38.634 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify stray on activate = true 2024-04-19T01:41:38.634 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug trim objects = true 2024-04-19T01:41:38.634 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-19T01:41:38.634 INFO:teuthology.orchestra.run.smithi071.stdout: osd open classes on start = true 2024-04-19T01:41:38.634 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug pg log writeout = true 2024-04-19T01:41:38.634 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-19T01:41:38.634 INFO:teuthology.orchestra.run.smithi071.stdout: osd deep scrub update digest min age = 30 2024-04-19T01:41:38.634 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-19T01:41:38.634 INFO:teuthology.orchestra.run.smithi071.stdout: osd map max advance = 10 2024-04-19T01:41:38.635 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-19T01:41:38.635 INFO:teuthology.orchestra.run.smithi071.stdout: journal zero on create = true 2024-04-19T01:41:38.635 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-19T01:41:38.635 INFO:teuthology.orchestra.run.smithi071.stdout: filestore ondisk finisher threads = 3 2024-04-19T01:41:38.635 INFO:teuthology.orchestra.run.smithi071.stdout: filestore apply finisher threads = 3 2024-04-19T01:41:38.635 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-19T01:41:38.635 INFO:teuthology.orchestra.run.smithi071.stdout: bdev debug aio = true 2024-04-19T01:41:38.635 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug misdirected ops = true 2024-04-19T01:41:38.635 INFO:teuthology.orchestra.run.smithi071.stdout: bdev async discard = True 2024-04-19T01:41:38.635 INFO:teuthology.orchestra.run.smithi071.stdout: bdev enable discard = True 2024-04-19T01:41:38.635 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore allocator = bitmap 2024-04-19T01:41:38.635 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore block size = 96636764160 2024-04-19T01:41:38.635 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore fsck on mount = True 2024-04-19T01:41:38.635 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore zero block detection = True 2024-04-19T01:41:38.636 INFO:teuthology.orchestra.run.smithi071.stdout: debug bluefs = 20 2024-04-19T01:41:38.636 INFO:teuthology.orchestra.run.smithi071.stdout: debug bluestore = 20 2024-04-19T01:41:38.636 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-04-19T01:41:38.636 INFO:teuthology.orchestra.run.smithi071.stdout: debug osd = 20 2024-04-19T01:41:38.636 INFO:teuthology.orchestra.run.smithi071.stdout: debug rocksdb = 10 2024-04-19T01:41:38.636 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd backfillfull_ratio = 0.85 2024-04-19T01:41:38.636 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd full ratio = 0.9 2024-04-19T01:41:38.636 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd nearfull ratio = 0.8 2024-04-19T01:41:38.636 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify cached snaps = True 2024-04-19T01:41:38.636 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify missing on start = True 2024-04-19T01:41:38.636 INFO:teuthology.orchestra.run.smithi071.stdout: osd failsafe full ratio = 0.95 2024-04-19T01:41:38.636 INFO:teuthology.orchestra.run.smithi071.stdout: osd mclock override recovery settings = True 2024-04-19T01:41:38.636 INFO:teuthology.orchestra.run.smithi071.stdout: osd objectstore = bluestore 2024-04-19T01:41:38.636 INFO:teuthology.orchestra.run.smithi071.stdout: osd op queue = debug_random 2024-04-19T01:41:38.636 INFO:teuthology.orchestra.run.smithi071.stdout: osd op queue cut off = debug_random 2024-04-19T01:41:38.637 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-19T01:41:38.637 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr] 2024-04-19T01:41:38.637 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-04-19T01:41:38.637 INFO:teuthology.orchestra.run.smithi071.stdout: debug mgr = 20 2024-04-19T01:41:38.637 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2024-04-19T01:41:38.637 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2024-04-19T01:41:38.637 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2024-04-19T01:41:38.637 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2024-04-19T01:41:38.637 INFO:teuthology.orchestra.run.smithi071.stdout: mgr/telemetry/nag = false 2024-04-19T01:41:38.637 INFO:teuthology.orchestra.run.smithi071.stdout: debug monc = 10 2024-04-19T01:41:38.637 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-19T01:41:38.637 INFO:teuthology.orchestra.run.smithi071.stdout:[mon] 2024-04-19T01:41:38.637 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-04-19T01:41:38.638 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2024-04-19T01:41:38.638 INFO:teuthology.orchestra.run.smithi071.stdout: debug paxos = 20 2024-04-19T01:41:38.638 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2024-04-19T01:41:38.638 INFO:teuthology.orchestra.run.smithi071.stdout: mon data avail warn = 5 2024-04-19T01:41:38.638 INFO:teuthology.orchestra.run.smithi071.stdout: mon mgr mkfs grace = 240 2024-04-19T01:41:38.638 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2024-04-19T01:41:38.638 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd reporter subtree level = osd 2024-04-19T01:41:38.638 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd prime pg temp = true 2024-04-19T01:41:38.638 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2024-04-19T01:41:38.638 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-19T01:41:38.638 INFO:teuthology.orchestra.run.smithi071.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-19T01:41:38.638 INFO:teuthology.orchestra.run.smithi071.stdout: auth mon ticket ttl = 660 # 11m 2024-04-19T01:41:38.638 INFO:teuthology.orchestra.run.smithi071.stdout: auth service ticket ttl = 240 # 4m 2024-04-19T01:41:38.638 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-19T01:41:38.638 INFO:teuthology.orchestra.run.smithi071.stdout: # don't complain about insecure global_id in the test suite 2024-04-19T01:41:38.639 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-19T01:41:38.639 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-19T01:41:38.639 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-19T01:41:38.639 INFO:teuthology.orchestra.run.smithi071.stdout: # 1m isn't quite enough 2024-04-19T01:41:38.639 INFO:teuthology.orchestra.run.smithi071.stdout: mon_down_mkfs_grace = 2m 2024-04-19T01:41:38.639 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-19T01:41:38.639 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_filestore_osds = false 2024-04-19T01:41:38.639 INFO:teuthology.orchestra.run.smithi071.stdout: mon min osdmap epochs = 25 2024-04-19T01:41:38.639 INFO:teuthology.orchestra.run.smithi071.stdout: mon osdmap full prune interval = 2 2024-04-19T01:41:38.639 INFO:teuthology.orchestra.run.smithi071.stdout: mon osdmap full prune min = 15 2024-04-19T01:41:38.639 INFO:teuthology.orchestra.run.smithi071.stdout: mon osdmap full prune txsize = 2 2024-04-19T01:41:38.639 INFO:teuthology.orchestra.run.smithi071.stdout: mon scrub inject crc mismatch = 0.01 2024-04-19T01:41:38.639 INFO:teuthology.orchestra.run.smithi071.stdout: mon scrub inject missing keys = 0.05 2024-04-19T01:41:38.639 INFO:teuthology.orchestra.run.smithi071.stdout: mon scrub interval = 300 2024-04-19T01:41:38.639 INFO:teuthology.orchestra.run.smithi071.stdout: paxos min = 10 2024-04-19T01:41:38.639 INFO:teuthology.orchestra.run.smithi071.stdout: paxos service trim min = 5 2024-04-19T01:41:38.640 INFO:teuthology.orchestra.run.smithi071.stdout: paxos trim min = 10 2024-04-19T01:41:38.640 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-19T01:41:38.640 INFO:teuthology.orchestra.run.smithi071.stdout:[client] 2024-04-19T01:41:38.640 INFO:teuthology.orchestra.run.smithi071.stdout: rgw cache enabled = true 2024-04-19T01:41:38.640 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable ops log = true 2024-04-19T01:41:38.640 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable usage log = true 2024-04-19T01:41:38.640 INFO:teuthology.orchestra.run.smithi071.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-19T01:41:38.640 INFO:teuthology.orchestra.run.smithi071.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-19T01:41:38.640 INFO:teuthology.orchestra.run.smithi071.stdout: debug monc = 20 2024-04-19T01:41:38.640 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-04-19T01:41:38.640 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.a] 2024-04-19T01:41:38.640 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.c] 2024-04-19T01:41:38.640 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.b] 2024-04-19T01:41:38.640 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-19T01:41:38.641 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-19T01:41:38.720 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-19T01:41:38.720 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-19T01:41:38.720 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-19T01:41:38.725 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-19T01:41:38.725 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-19T01:41:38.775 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi062.front.sepia.ceph.com 2024-04-19T01:41:38.775 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-19T01:41:38.775 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-19T01:41:38.775 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-19T01:41:38.841 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-19T01:41:38.841 DEBUG:teuthology.orchestra.run.smithi062:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:41:38.891 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi071.front.sepia.ceph.com 2024-04-19T01:41:38.891 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-19T01:41:38.891 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-19T01:41:38.891 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-19T01:41:38.914 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-19T01:41:38.914 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:41:38.960 INFO:tasks.ceph:Setting up mon nodes... 2024-04-19T01:41:38.960 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-19T01:41:38.960 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-19T01:41:39.044 INFO:teuthology.orchestra.run.smithi071.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-19T01:41:39.071 INFO:tasks.ceph:Setting up mds nodes... 2024-04-19T01:41:39.071 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-19T01:41:39.071 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-19T01:41:39.146 INFO:teuthology.orchestra.run.smithi071.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-19T01:41:39.156 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-19T01:41:39.157 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi062.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi071.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-04-19T01:41:39.157 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-19T01:41:39.171 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-19T01:41:39.171 INFO:tasks.ceph:role: osd.0 2024-04-19T01:41:39.171 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi062.front.sepia.ceph.com 2024-04-19T01:41:39.171 DEBUG:teuthology.orchestra.run.smithi062:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-19T01:41:39.234 INFO:teuthology.orchestra.run.smithi062.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T01:41:39.234 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T01:41:39.234 INFO:teuthology.orchestra.run.smithi062.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T01:41:39.235 INFO:teuthology.orchestra.run.smithi062.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T01:41:39.235 INFO:teuthology.orchestra.run.smithi062.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T01:41:39.235 INFO:teuthology.orchestra.run.smithi062.stdout: = sunit=0 swidth=0 blks 2024-04-19T01:41:39.235 INFO:teuthology.orchestra.run.smithi062.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T01:41:39.235 INFO:teuthology.orchestra.run.smithi062.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T01:41:39.235 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T01:41:39.235 INFO:teuthology.orchestra.run.smithi062.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T01:41:39.697 INFO:teuthology.orchestra.run.smithi062.stdout:Discarding blocks...Done. 2024-04-19T01:41:39.699 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi062.front.sepia.ceph.com -o noatime 2024-04-19T01:41:39.699 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-19T01:41:39.791 DEBUG:teuthology.orchestra.run.smithi062:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-19T01:41:39.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:41:39.800 INFO:teuthology.orchestra.run.smithi062.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T01:41:39.801 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-19T01:41:39.852 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-19T01:41:39.852 INFO:tasks.ceph:role: osd.1 2024-04-19T01:41:39.852 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi062.front.sepia.ceph.com 2024-04-19T01:41:39.852 DEBUG:teuthology.orchestra.run.smithi062:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-19T01:41:39.918 INFO:teuthology.orchestra.run.smithi062.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T01:41:39.918 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T01:41:39.918 INFO:teuthology.orchestra.run.smithi062.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T01:41:39.918 INFO:teuthology.orchestra.run.smithi062.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T01:41:39.918 INFO:teuthology.orchestra.run.smithi062.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T01:41:39.918 INFO:teuthology.orchestra.run.smithi062.stdout: = sunit=0 swidth=0 blks 2024-04-19T01:41:39.918 INFO:teuthology.orchestra.run.smithi062.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T01:41:39.918 INFO:teuthology.orchestra.run.smithi062.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T01:41:39.918 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T01:41:39.918 INFO:teuthology.orchestra.run.smithi062.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T01:41:40.390 INFO:teuthology.orchestra.run.smithi062.stdout:Discarding blocks...Done. 2024-04-19T01:41:40.392 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi062.front.sepia.ceph.com -o noatime 2024-04-19T01:41:40.393 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-19T01:41:40.539 DEBUG:teuthology.orchestra.run.smithi062:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-19T01:41:40.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:41:40.551 INFO:teuthology.orchestra.run.smithi062.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T01:41:40.551 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-19T01:41:40.605 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-19T01:41:40.606 INFO:tasks.ceph:role: osd.2 2024-04-19T01:41:40.606 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi062.front.sepia.ceph.com 2024-04-19T01:41:40.606 DEBUG:teuthology.orchestra.run.smithi062:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-19T01:41:40.670 INFO:teuthology.orchestra.run.smithi062.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T01:41:40.670 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T01:41:40.670 INFO:teuthology.orchestra.run.smithi062.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T01:41:40.670 INFO:teuthology.orchestra.run.smithi062.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T01:41:40.670 INFO:teuthology.orchestra.run.smithi062.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T01:41:40.670 INFO:teuthology.orchestra.run.smithi062.stdout: = sunit=0 swidth=0 blks 2024-04-19T01:41:40.670 INFO:teuthology.orchestra.run.smithi062.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T01:41:40.670 INFO:teuthology.orchestra.run.smithi062.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T01:41:40.670 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T01:41:40.670 INFO:teuthology.orchestra.run.smithi062.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T01:41:41.135 INFO:teuthology.orchestra.run.smithi062.stdout:Discarding blocks...Done. 2024-04-19T01:41:41.137 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi062.front.sepia.ceph.com -o noatime 2024-04-19T01:41:41.137 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-19T01:41:41.285 DEBUG:teuthology.orchestra.run.smithi062:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-19T01:41:41.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:41:41.296 INFO:teuthology.orchestra.run.smithi062.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T01:41:41.296 DEBUG:teuthology.orchestra.run.smithi062:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:41:41.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:41:41.406+0000 7f87b7f3f7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-19T01:41:41.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:41:41.406+0000 7f87b7f3f7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-19T01:41:41.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:41:41.406+0000 7f87b7f3f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-19T01:41:41.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:41:41.406+0000 7f87b7f3f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-19T01:41:41.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:41:41.406+0000 7f87b7f3f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-19T01:41:41.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:41:41.410+0000 7f87b7f3f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-19T01:41:44.379 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-19T01:41:44.393 DEBUG:teuthology.orchestra.run.smithi062:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:41:44.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:41:44.474+0000 7f7ba68087c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-19T01:41:44.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:41:44.474+0000 7f7ba68087c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-19T01:41:44.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:41:44.474+0000 7f7ba68087c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-19T01:41:44.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:41:44.474+0000 7f7ba68087c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-19T01:41:44.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:41:44.474+0000 7f7ba68087c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-19T01:41:44.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:41:44.474+0000 7f7ba68087c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-19T01:41:47.442 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-19T01:41:47.499 DEBUG:teuthology.orchestra.run.smithi062:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:41:47.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:41:47.582+0000 7f6c330897c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-19T01:41:47.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:41:47.582+0000 7f6c330897c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-19T01:41:47.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:41:47.582+0000 7f6c330897c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-19T01:41:47.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:41:47.582+0000 7f6c330897c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-19T01:41:47.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:41:47.582+0000 7f6c330897c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-19T01:41:47.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:41:47.582+0000 7f6c330897c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-19T01:41:50.550 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-19T01:41:50.606 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-19T01:41:50.621 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-19T01:41:50.621 INFO:tasks.ceph:role: osd.3 2024-04-19T01:41:50.621 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi071.front.sepia.ceph.com 2024-04-19T01:41:50.621 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-19T01:41:50.684 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T01:41:50.684 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T01:41:50.684 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T01:41:50.684 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T01:41:50.685 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T01:41:50.685 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-04-19T01:41:50.685 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T01:41:50.685 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T01:41:50.685 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T01:41:50.685 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T01:41:51.174 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-04-19T01:41:51.176 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-04-19T01:41:51.177 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-04-19T01:41:51.315 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-19T01:41:51.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:41:51.327 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T01:41:51.327 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-19T01:41:51.384 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-19T01:41:51.384 INFO:tasks.ceph:role: osd.4 2024-04-19T01:41:51.384 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi071.front.sepia.ceph.com 2024-04-19T01:41:51.384 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-19T01:41:51.447 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T01:41:51.447 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T01:41:51.447 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T01:41:51.447 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T01:41:51.447 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T01:41:51.447 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-04-19T01:41:51.447 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T01:41:51.447 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T01:41:51.448 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T01:41:51.448 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T01:41:51.878 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-04-19T01:41:51.880 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-04-19T01:41:51.881 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-04-19T01:41:51.943 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-19T01:41:51.993 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T01:41:51.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:41:51.994 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-19T01:41:52.051 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-19T01:41:52.051 INFO:tasks.ceph:role: osd.5 2024-04-19T01:41:52.051 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi071.front.sepia.ceph.com 2024-04-19T01:41:52.051 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-19T01:41:52.115 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T01:41:52.115 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T01:41:52.115 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T01:41:52.115 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T01:41:52.115 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T01:41:52.115 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-04-19T01:41:52.115 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T01:41:52.115 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T01:41:52.115 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T01:41:52.116 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T01:41:52.590 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-04-19T01:41:52.592 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-04-19T01:41:52.592 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-04-19T01:41:52.693 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-19T01:41:52.702 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T01:41:52.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:41:52.703 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:41:52.811 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:41:52.809+0000 7f9bc4d537c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-19T01:41:52.814 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:41:52.813+0000 7f9bc4d537c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-19T01:41:52.815 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:41:52.813+0000 7f9bc4d537c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-19T01:41:52.815 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:41:52.813+0000 7f9bc4d537c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-19T01:41:52.815 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:41:52.813+0000 7f9bc4d537c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-19T01:41:52.820 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:41:52.825+0000 7f9bc4d537c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-19T01:41:55.800 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-19T01:41:55.813 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:41:55.894 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:41:55.897+0000 7f2f6e3097c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-19T01:41:55.895 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:41:55.897+0000 7f2f6e3097c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-19T01:41:55.895 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:41:55.897+0000 7f2f6e3097c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-19T01:41:55.895 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:41:55.897+0000 7f2f6e3097c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-19T01:41:55.895 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:41:55.897+0000 7f2f6e3097c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-19T01:41:55.895 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:41:55.897+0000 7f2f6e3097c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-19T01:41:58.890 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-19T01:41:58.946 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:41:59.030 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:41:59.033+0000 7f3b3d14b7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-19T01:41:59.031 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:41:59.033+0000 7f3b3d14b7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-19T01:41:59.031 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:41:59.033+0000 7f3b3d14b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-19T01:41:59.031 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:41:59.033+0000 7f3b3d14b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-19T01:41:59.031 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:41:59.033+0000 7f3b3d14b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-19T01:41:59.032 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:41:59.037+0000 7f3b3d14b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-19T01:42:02.057 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-19T01:42:02.112 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-19T01:42:02.112 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-19T01:42:02.112 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-19T01:42:02.127 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-19T01:42:02.127 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-19T01:42:02.181 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-19T01:42:02.181 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-19T01:42:02.237 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-19T01:42:02.237 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-19T01:42:02.250 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-19T01:42:02.250 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-19T01:42:02.303 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-19T01:42:02.303 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-19T01:42:02.360 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-19T01:42:02.360 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-19T01:42:02.415 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-19T01:42:02.415 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-19T01:42:02.464 INFO:tasks.ceph:Adding keys to all mons... 2024-04-19T01:42:02.464 DEBUG:teuthology.orchestra.run.smithi062:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-19T01:42:02.466 DEBUG:teuthology.orchestra.run.smithi071:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-19T01:42:02.510 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.0] 2024-04-19T01:42:02.510 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQDVyyFmRdGjGBAAbrF8WA/n9itA1qQ6vYXwgA== 2024-04-19T01:42:02.510 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.1] 2024-04-19T01:42:02.510 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQDYyyFm3IWBHBAAEma9O9a9qhstfeJNlxTvpg== 2024-04-19T01:42:02.510 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.2] 2024-04-19T01:42:02.510 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQDbyyFm0/37IhAAdSmSoSdhpOd9C5SMjcDKMw== 2024-04-19T01:42:02.510 INFO:teuthology.orchestra.run.smithi062.stdout:[mgr.x] 2024-04-19T01:42:02.510 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQDTyyFmEXEFAxAArSTLJxQLL+cqP8L4evMdfA== 2024-04-19T01:42:02.510 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.3] 2024-04-19T01:42:02.510 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQDgyyFmnVmxMBAAlde8zgLxsefvZpD+MjT4AQ== 2024-04-19T01:42:02.511 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.4] 2024-04-19T01:42:02.511 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQDjyyFmUhGsNRAARfwsOEB6jxP+MoojUZTsMQ== 2024-04-19T01:42:02.511 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.5] 2024-04-19T01:42:02.511 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQDnyyFmp+gvAhAAEpuDjWMVXrR7U9uWdh2P/A== 2024-04-19T01:42:02.511 INFO:teuthology.orchestra.run.smithi062.stdout:[client.0] 2024-04-19T01:42:02.511 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQDTyyFmiecZCRAAqLkxhmQePIrMp36pxH4DUg== 2024-04-19T01:42:02.518 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.0] 2024-04-19T01:42:02.518 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDVyyFmRdGjGBAAbrF8WA/n9itA1qQ6vYXwgA== 2024-04-19T01:42:02.518 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.1] 2024-04-19T01:42:02.518 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDYyyFm3IWBHBAAEma9O9a9qhstfeJNlxTvpg== 2024-04-19T01:42:02.518 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.2] 2024-04-19T01:42:02.518 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDbyyFm0/37IhAAdSmSoSdhpOd9C5SMjcDKMw== 2024-04-19T01:42:02.518 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr.x] 2024-04-19T01:42:02.518 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDTyyFmEXEFAxAArSTLJxQLL+cqP8L4evMdfA== 2024-04-19T01:42:02.519 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.3] 2024-04-19T01:42:02.519 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDgyyFmnVmxMBAAlde8zgLxsefvZpD+MjT4AQ== 2024-04-19T01:42:02.519 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.4] 2024-04-19T01:42:02.519 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDjyyFmUhGsNRAARfwsOEB6jxP+MoojUZTsMQ== 2024-04-19T01:42:02.519 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.5] 2024-04-19T01:42:02.519 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDnyyFmp+gvAhAAEpuDjWMVXrR7U9uWdh2P/A== 2024-04-19T01:42:02.519 INFO:teuthology.orchestra.run.smithi071.stdout:[client.0] 2024-04-19T01:42:02.519 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDTyyFmiecZCRAAqLkxhmQePIrMp36pxH4DUg== 2024-04-19T01:42:02.520 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:42:02.555 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:42:02.607 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:42:02.609 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:42:02.695 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:42:02.697 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:42:02.783 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-19T01:42:02.785 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-19T01:42:02.871 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:42:02.876 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:42:02.966 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:42:02.974 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:42:03.018 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:42:03.065 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T01:42:03.109 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-19T01:42:03.156 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-19T01:42:03.206 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-19T01:42:03.206 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-19T01:42:03.255 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-19T01:42:03.355 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-19T01:42:03.369 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-19T01:42:03.427 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-19T01:42:03.523 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-19T01:42:03.535 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-19T01:42:03.550 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-19T01:42:03.650 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-19T01:42:03.663 DEBUG:teuthology.orchestra.run.smithi062:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:42:03.667 DEBUG:teuthology.orchestra.run.smithi071:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-19T01:42:03.708 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-19T01:42:03.708 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-19T01:42:03.709 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-19T01:42:03.716 INFO:tasks.ceph.mon.a:Started 2024-04-19T01:42:03.716 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-19T01:42:03.716 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-19T01:42:03.719 INFO:tasks.ceph.mon.c:Started 2024-04-19T01:42:03.720 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-19T01:42:03.720 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-19T01:42:03.754 INFO:tasks.ceph.mon.b:Started 2024-04-19T01:42:03.754 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-19T01:42:03.755 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-19T01:42:03.755 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-19T01:42:03.758 INFO:tasks.ceph.mgr.x:Started 2024-04-19T01:42:03.758 INFO:tasks.ceph:Setting crush tunables to default 2024-04-19T01:42:03.759 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster ceph osd crush tunables default 2024-04-19T01:42:03.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.874+0000 7fdfb70ef640 10 monclient: get_monmap_and_config 2024-04-19T01:42:03.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.874+0000 7fdfb70ef640 10 monclient: build_initial_monmap 2024-04-19T01:42:03.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfb70ef640 10 monclient: monmap: 2024-04-19T01:42:03.878 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:03.878 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:03.878 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:03.882041+0000 2024-04-19T01:42:03.878 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:03.882041+0000 2024-04-19T01:42:03.879 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:03.879 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:03.879 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:03.879 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:03.879 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:03.879 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:03.879 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:03.879 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:03.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfb70ef640 1 Processor -- start 2024-04-19T01:42:03.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfb70ef640 1 -- start start 2024-04-19T01:42:03.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfb70ef640 10 monclient: init 2024-04-19T01:42:03.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfb70ef640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:03.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfb70ef640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-04-19T01:42:03.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfb70ef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdfb0137780 addr v1:172.21.15.71:6789/0 2024-04-19T01:42:03.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfb70ef640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdfb00a60b0 0x7fdfb013c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:03.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfb70ef640 10 monclient(hunting): picked mon.noname-a con 0x7fdfb00a60b0 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:03.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfb70ef640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fdfb013cc60 0x7fdfb013f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:03.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfb70ef640 10 monclient(hunting): picked mon.noname-c con 0x7fdfb013cc60 addr v2:172.21.15.71:3300/0 2024-04-19T01:42:03.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfb70ef640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfb009c2e0 con 0x7fdfb0137780 2024-04-19T01:42:03.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfb70ef640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:03.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfb70ef640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fdfb009bd20 con 0x7fdfb00a60b0 2024-04-19T01:42:03.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfb70ef640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:03.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfb70ef640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fdfb009c000 con 0x7fdfb013cc60 2024-04-19T01:42:03.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfb70ef640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:03.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfb70ef640 10 monclient(hunting): authenticate will time out at 909.701137s 2024-04-19T01:42:03.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfaffff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdfb00a60b0 0x7fdfb013c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:03.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfaffff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdfb00a60b0 0x7fdfb013c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41112/0 (socket says 172.21.15.62:41112) 2024-04-19T01:42:03.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfaffff640 1 -- 172.21.15.62:0/698029401 learned_addr learned my addr 172.21.15.62:0/698029401 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:03.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfaffff640 10 monclient(hunting): get_auth_request con 0x7fdfb00a60b0 auth_method 0 2024-04-19T01:42:03.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:03.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfaffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:03.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfaffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:03.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfaffff640 1 -- 172.21.15.62:0/698029401 >> v2:172.21.15.62:3300/0 conn(0x7fdfb00a60b0 msgr2=0x7fdfb013c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:42:03.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfaffff640 1 -- 172.21.15.62:0/698029401 >> v2:172.21.15.62:3300/0 conn(0x7fdfb00a60b0 msgr2=0x7fdfb013c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:42:03.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfaffff640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.62:3300/0 conn(0x7fdfb00a60b0 0x7fdfb013c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:42:03.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfaffff640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.62:3300/0 conn(0x7fdfb00a60b0 0x7fdfb013c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:42:03.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfb568c640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.71:3300/0 conn(0x7fdfb013cc60 0x7fdfb013f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:03.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfb568c640 10 monclient(hunting): get_auth_request con 0x7fdfb013cc60 auth_method 0 2024-04-19T01:42:03.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfb568c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:03.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfb568c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:03.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfb568c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:03.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfb568c640 1 -- 172.21.15.62:0/698029401 >> v2:172.21.15.71:3300/0 conn(0x7fdfb013cc60 msgr2=0x7fdfb013f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:42:03.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfb568c640 1 -- 172.21.15.62:0/698029401 >> v2:172.21.15.71:3300/0 conn(0x7fdfb013cc60 msgr2=0x7fdfb013f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:42:03.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfb568c640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.71:3300/0 conn(0x7fdfb013cc60 0x7fdfb013f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:42:03.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfb568c640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.71:3300/0 conn(0x7fdfb013cc60 0x7fdfb013f050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:42:03.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:03.878+0000 7fdfb4e8b640 1 -- 172.21.15.62:0/698029401 >> v1:172.21.15.71:6789/0 conn(0x7fdfb0137780 legacy=0x7fdfb0137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:04.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.082+0000 7fdfaffff640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.62:3300/0 conn(0x7fdfb00a60b0 0x7fdfb013c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:04.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.082+0000 7fdfaffff640 10 monclient(hunting): get_auth_request con 0x7fdfb00a60b0 auth_method 0 2024-04-19T01:42:04.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.082+0000 7fdfaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:04.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.082+0000 7fdfaffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:04.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.082+0000 7fdfaffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:42:04.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.082+0000 7fdfb568c640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.71:3300/0 conn(0x7fdfb013cc60 0x7fdfb013f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:04.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.082+0000 7fdfb568c640 10 monclient(hunting): get_auth_request con 0x7fdfb013cc60 auth_method 0 2024-04-19T01:42:04.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.082+0000 7fdfb568c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:04.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.082+0000 7fdfb568c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:04.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.082+0000 7fdfb568c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:42:04.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.082+0000 7fdfaffff640 1 -- 172.21.15.62:0/698029401 >> v2:172.21.15.62:3300/0 conn(0x7fdfb00a60b0 msgr2=0x7fdfb013c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:42:04.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.082+0000 7fdfaffff640 1 -- 172.21.15.62:0/698029401 >> v2:172.21.15.62:3300/0 conn(0x7fdfb00a60b0 msgr2=0x7fdfb013c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:42:04.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.082+0000 7fdfaffff640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.62:3300/0 conn(0x7fdfb00a60b0 0x7fdfb013c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:42:04.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.082+0000 7fdfaffff640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.62:3300/0 conn(0x7fdfb00a60b0 0x7fdfb013c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:42:04.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.082+0000 7fdfb568c640 1 -- 172.21.15.62:0/698029401 >> v2:172.21.15.71:3300/0 conn(0x7fdfb013cc60 msgr2=0x7fdfb013f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:42:04.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.082+0000 7fdfb568c640 1 -- 172.21.15.62:0/698029401 >> v2:172.21.15.71:3300/0 conn(0x7fdfb013cc60 msgr2=0x7fdfb013f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:42:04.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.082+0000 7fdfb568c640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.71:3300/0 conn(0x7fdfb013cc60 0x7fdfb013f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:42:04.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.082+0000 7fdfb568c640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.71:3300/0 conn(0x7fdfb013cc60 0x7fdfb013f050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:42:04.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.486+0000 7fdfaffff640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.62:3300/0 conn(0x7fdfb00a60b0 0x7fdfb013c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:04.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.486+0000 7fdfaffff640 10 monclient(hunting): get_auth_request con 0x7fdfb00a60b0 auth_method 0 2024-04-19T01:42:04.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.486+0000 7fdfaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:04.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.486+0000 7fdfaffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:04.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.486+0000 7fdfaffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:42:04.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.486+0000 7fdfb568c640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.71:3300/0 conn(0x7fdfb013cc60 0x7fdfb013f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:04.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.486+0000 7fdfb568c640 10 monclient(hunting): get_auth_request con 0x7fdfb013cc60 auth_method 0 2024-04-19T01:42:04.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.486+0000 7fdfb568c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:04.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.486+0000 7fdfb568c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:04.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.486+0000 7fdfb568c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:42:04.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.486+0000 7fdfaffff640 1 -- 172.21.15.62:0/698029401 >> v2:172.21.15.62:3300/0 conn(0x7fdfb00a60b0 msgr2=0x7fdfb013c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:42:04.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.486+0000 7fdfaffff640 1 -- 172.21.15.62:0/698029401 >> v2:172.21.15.62:3300/0 conn(0x7fdfb00a60b0 msgr2=0x7fdfb013c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:42:04.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.486+0000 7fdfaffff640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.62:3300/0 conn(0x7fdfb00a60b0 0x7fdfb013c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:42:04.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.486+0000 7fdfaffff640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.62:3300/0 conn(0x7fdfb00a60b0 0x7fdfb013c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:42:04.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.486+0000 7fdfb568c640 1 -- 172.21.15.62:0/698029401 >> v2:172.21.15.71:3300/0 conn(0x7fdfb013cc60 msgr2=0x7fdfb013f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:42:04.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.486+0000 7fdfb568c640 1 -- 172.21.15.62:0/698029401 >> v2:172.21.15.71:3300/0 conn(0x7fdfb013cc60 msgr2=0x7fdfb013f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:42:04.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.486+0000 7fdfb568c640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.71:3300/0 conn(0x7fdfb013cc60 0x7fdfb013f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:42:04.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:04.486+0000 7fdfb568c640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.71:3300/0 conn(0x7fdfb013cc60 0x7fdfb013f050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:42:05.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:05.290+0000 7fdfaffff640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.62:3300/0 conn(0x7fdfb00a60b0 0x7fdfb013c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:05.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:05.290+0000 7fdfaffff640 10 monclient(hunting): get_auth_request con 0x7fdfb00a60b0 auth_method 0 2024-04-19T01:42:05.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:05.290+0000 7fdfaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:05.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:05.290+0000 7fdfaffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:05.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:05.290+0000 7fdfaffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:42:05.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:05.290+0000 7fdfb568c640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.71:3300/0 conn(0x7fdfb013cc60 0x7fdfb013f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:05.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:05.290+0000 7fdfb568c640 10 monclient(hunting): get_auth_request con 0x7fdfb013cc60 auth_method 0 2024-04-19T01:42:05.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:05.290+0000 7fdfb568c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:05.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:05.290+0000 7fdfb568c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:05.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:05.290+0000 7fdfb568c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:42:05.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:05.290+0000 7fdfaffff640 1 -- 172.21.15.62:0/698029401 >> v2:172.21.15.62:3300/0 conn(0x7fdfb00a60b0 msgr2=0x7fdfb013c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:42:05.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:05.290+0000 7fdfaffff640 1 -- 172.21.15.62:0/698029401 >> v2:172.21.15.62:3300/0 conn(0x7fdfb00a60b0 msgr2=0x7fdfb013c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:42:05.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:05.290+0000 7fdfaffff640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.62:3300/0 conn(0x7fdfb00a60b0 0x7fdfb013c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:42:05.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:05.290+0000 7fdfaffff640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.62:3300/0 conn(0x7fdfb00a60b0 0x7fdfb013c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:42:05.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:05.290+0000 7fdfb568c640 1 -- 172.21.15.62:0/698029401 >> v2:172.21.15.71:3300/0 conn(0x7fdfb013cc60 msgr2=0x7fdfb013f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:42:05.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:05.290+0000 7fdfb568c640 1 -- 172.21.15.62:0/698029401 >> v2:172.21.15.71:3300/0 conn(0x7fdfb013cc60 msgr2=0x7fdfb013f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:42:05.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:05.290+0000 7fdfb568c640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.71:3300/0 conn(0x7fdfb013cc60 0x7fdfb013f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:42:05.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:05.290+0000 7fdfb568c640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.71:3300/0 conn(0x7fdfb013cc60 0x7fdfb013f050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:42:06.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:06.882+0000 7fdfae7fc640 10 monclient(hunting): tick 2024-04-19T01:42:06.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:06.882+0000 7fdfae7fc640 10 monclient(hunting): _check_auth_tickets 2024-04-19T01:42:06.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:06.882+0000 7fdfae7fc640 1 monclient(hunting): continuing hunt 2024-04-19T01:42:06.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:06.882+0000 7fdfae7fc640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T01:42:06.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:06.882+0000 7fdfae7fc640 1 -- 172.21.15.62:0/698029401 >> v2:172.21.15.71:3300/0 conn(0x7fdfb013cc60 msgr2=0x7fdfb013f050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:42:06.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:06.882+0000 7fdfae7fc640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.71:3300/0 conn(0x7fdfb013cc60 0x7fdfb013f050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:06.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:06.882+0000 7fdfae7fc640 1 -- 172.21.15.62:0/698029401 >> v2:172.21.15.62:3300/0 conn(0x7fdfb00a60b0 msgr2=0x7fdfb013c720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:42:06.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:06.882+0000 7fdfae7fc640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.62:3300/0 conn(0x7fdfb00a60b0 0x7fdfb013c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:06.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:06.882+0000 7fdfae7fc640 1 -- 172.21.15.62:0/698029401 >> v1:172.21.15.71:6789/0 conn(0x7fdfb0137780 legacy=0x7fdfb0137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:06.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:06.882+0000 7fdfae7fc640 10 monclient: _add_conns ranks=[0,2] 2024-04-19T01:42:06.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:06.882+0000 7fdfae7fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdf9c003720 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:06.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:06.882+0000 7fdfae7fc640 1 --2- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf9c007350 0x7fdf9c009810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:06.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:06.882+0000 7fdfae7fc640 10 monclient(hunting): picked mon.noname-b con 0x7fdf9c007350 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:06.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:06.882+0000 7fdfae7fc640 1 -- 172.21.15.62:0/698029401 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfb0138270 con 0x7fdf9c003720 2024-04-19T01:42:06.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:06.882+0000 7fdfae7fc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:06.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:06.882+0000 7fdfae7fc640 1 -- 172.21.15.62:0/698029401 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fdfb009bd20 con 0x7fdf9c007350 2024-04-19T01:42:06.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:06.882+0000 7fdfae7fc640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:06.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:06.882+0000 7fdfaffff640 1 --2- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf9c007350 0x7fdf9c009810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:06.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:06.882+0000 7fdfaffff640 10 monclient(hunting): get_auth_request con 0x7fdf9c007350 auth_method 0 2024-04-19T01:42:06.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:06.882+0000 7fdfaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:06.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:06.882+0000 7fdfaffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:06.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:06.882+0000 7fdfaffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:06.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:06.882+0000 7fdfaffff640 1 -- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf9c007350 msgr2=0x7fdf9c009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:42:06.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:06.882+0000 7fdfaffff640 1 -- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf9c007350 msgr2=0x7fdf9c009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:42:06.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:06.882+0000 7fdfaffff640 1 --2- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf9c007350 0x7fdf9c009810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:42:06.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:06.882+0000 7fdfaffff640 1 --2- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf9c007350 0x7fdf9c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:42:06.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:06.882+0000 7fdfb4e8b640 1 -- 172.21.15.62:0/698029401 >> v1:172.21.15.62:6789/0 conn(0x7fdf9c003720 legacy=0x7fdf9c005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:07.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:07.086+0000 7fdfaffff640 1 --2- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf9c007350 0x7fdf9c009810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:07.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:07.086+0000 7fdfaffff640 10 monclient(hunting): get_auth_request con 0x7fdf9c007350 auth_method 0 2024-04-19T01:42:07.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:07.086+0000 7fdfaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:07.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:07.086+0000 7fdfaffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:07.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:07.086+0000 7fdfaffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:42:07.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:07.086+0000 7fdfaffff640 1 -- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf9c007350 msgr2=0x7fdf9c009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:42:07.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:07.086+0000 7fdfaffff640 1 -- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf9c007350 msgr2=0x7fdf9c009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:42:07.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:07.086+0000 7fdfaffff640 1 --2- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf9c007350 0x7fdf9c009810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:42:07.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:07.086+0000 7fdfaffff640 1 --2- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf9c007350 0x7fdf9c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:42:07.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:07.490+0000 7fdfaffff640 1 --2- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf9c007350 0x7fdf9c009810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:07.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:07.490+0000 7fdfaffff640 10 monclient(hunting): get_auth_request con 0x7fdf9c007350 auth_method 0 2024-04-19T01:42:07.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:07.490+0000 7fdfaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:07.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:07.490+0000 7fdfaffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:07.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:07.490+0000 7fdfaffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:42:07.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:07.490+0000 7fdfaffff640 1 -- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf9c007350 msgr2=0x7fdf9c009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:42:07.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:07.490+0000 7fdfaffff640 1 -- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf9c007350 msgr2=0x7fdf9c009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:42:07.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:07.490+0000 7fdfaffff640 1 --2- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf9c007350 0x7fdf9c009810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:42:07.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:07.490+0000 7fdfaffff640 1 --2- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf9c007350 0x7fdf9c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:42:08.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:08.294+0000 7fdfaffff640 1 --2- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf9c007350 0x7fdf9c009810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:08.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:08.294+0000 7fdfaffff640 10 monclient(hunting): get_auth_request con 0x7fdf9c007350 auth_method 0 2024-04-19T01:42:08.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:08.294+0000 7fdfaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:08.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:08.294+0000 7fdfaffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:08.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:08.294+0000 7fdfaffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:42:08.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:08.294+0000 7fdfaffff640 1 -- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf9c007350 msgr2=0x7fdf9c009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:42:08.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:08.294+0000 7fdfaffff640 1 -- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf9c007350 msgr2=0x7fdf9c009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:42:08.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:08.294+0000 7fdfaffff640 1 --2- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf9c007350 0x7fdf9c009810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:42:08.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:08.294+0000 7fdfaffff640 1 --2- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf9c007350 0x7fdf9c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:42:09.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfae7fc640 10 monclient(hunting): tick 2024-04-19T01:42:09.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfae7fc640 10 monclient(hunting): _check_auth_tickets 2024-04-19T01:42:09.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfae7fc640 1 monclient(hunting): continuing hunt 2024-04-19T01:42:09.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfae7fc640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T01:42:09.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfae7fc640 1 -- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf9c007350 msgr2=0x7fdf9c009810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:42:09.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfae7fc640 1 --2- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf9c007350 0x7fdf9c009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:09.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfae7fc640 1 -- 172.21.15.62:0/698029401 >> v1:172.21.15.62:6789/0 conn(0x7fdf9c003720 legacy=0x7fdf9c005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:09.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfae7fc640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-04-19T01:42:09.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfae7fc640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.71:3300/0 conn(0x7fdfb013cc60 0x7fdf9c00a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:09.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfae7fc640 10 monclient(hunting): picked mon.noname-c con 0x7fdfb013cc60 addr v2:172.21.15.71:3300/0 2024-04-19T01:42:09.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfb568c640 1 -- 172.21.15.62:0/698029401 reap_dead start 2024-04-19T01:42:09.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfae7fc640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.62:3300/0 conn(0x7fdfb00a60b0 0x7fdf9c00a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:09.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfae7fc640 10 monclient(hunting): picked mon.noname-a con 0x7fdfb00a60b0 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:09.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfae7fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdf9c007800 addr v1:172.21.15.71:6789/0 2024-04-19T01:42:09.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfae7fc640 1 -- 172.21.15.62:0/698029401 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf9c00aff0 con 0x7fdf9c007800 2024-04-19T01:42:09.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfae7fc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:09.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfae7fc640 1 -- 172.21.15.62:0/698029401 --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fdfb009bd20 con 0x7fdfb00a60b0 2024-04-19T01:42:09.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfae7fc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:09.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfae7fc640 1 -- 172.21.15.62:0/698029401 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fdfb009c000 con 0x7fdfb013cc60 2024-04-19T01:42:09.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfae7fc640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:09.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfaffff640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.62:3300/0 conn(0x7fdfb00a60b0 0x7fdf9c00a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:09.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfaffff640 10 monclient(hunting): get_auth_request con 0x7fdfb00a60b0 auth_method 0 2024-04-19T01:42:09.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:09.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfaffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:09.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfaffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:09.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfb4e8b640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.71:3300/0 conn(0x7fdfb013cc60 0x7fdf9c00a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:09.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfb4e8b640 10 monclient(hunting): get_auth_request con 0x7fdfb013cc60 auth_method 0 2024-04-19T01:42:09.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfb4e8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:09.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfb4e8b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:09.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfb4e8b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:09.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfaffff640 1 -- 172.21.15.62:0/698029401 >> v2:172.21.15.62:3300/0 conn(0x7fdfb00a60b0 msgr2=0x7fdf9c00a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:42:09.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfaffff640 1 -- 172.21.15.62:0/698029401 >> v2:172.21.15.62:3300/0 conn(0x7fdfb00a60b0 msgr2=0x7fdf9c00a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:42:09.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfaffff640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.62:3300/0 conn(0x7fdfb00a60b0 0x7fdf9c00a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:42:09.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfaffff640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.62:3300/0 conn(0x7fdfb00a60b0 0x7fdf9c00a660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:42:09.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfb4e8b640 1 -- 172.21.15.62:0/698029401 >> v2:172.21.15.71:3300/0 conn(0x7fdfb013cc60 msgr2=0x7fdf9c00a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:42:09.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfb4e8b640 1 -- 172.21.15.62:0/698029401 >> v2:172.21.15.71:3300/0 conn(0x7fdfb013cc60 msgr2=0x7fdf9c00a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:42:09.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfb4e8b640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.71:3300/0 conn(0x7fdfb013cc60 0x7fdf9c00a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:42:09.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfb4e8b640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.71:3300/0 conn(0x7fdfb013cc60 0x7fdf9c00a100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:42:09.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.882+0000 7fdfb568c640 1 -- 172.21.15.62:0/698029401 >> v1:172.21.15.71:6789/0 conn(0x7fdf9c007800 legacy=0x7fdf9c003b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:09.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.910+0000 7fdfaf7fe640 1 -- 172.21.15.62:0/698029401 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 886319097 0 0) 0x7fdf9c00aff0 con 0x7fdf9c007800 2024-04-19T01:42:09.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.910+0000 7fdfaf7fe640 10 monclient(hunting): _negotiate 2024-04-19T01:42:09.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.910+0000 7fdfaf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:09.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.910+0000 7fdfaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:09.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.910+0000 7fdfaf7fe640 10 monclient(hunting): my global_id is 4099 2024-04-19T01:42:09.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.910+0000 7fdfaf7fe640 1 -- 172.21.15.62:0/698029401 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf98003530 con 0x7fdf9c007800 2024-04-19T01:42:09.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 1 -- 172.21.15.62:0/698029401 <== mon.1 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 929588657 0 0) 0x7fdf98003530 con 0x7fdf9c007800 2024-04-19T01:42:09.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 1 -- 172.21.15.62:0/698029401 >> v2:172.21.15.71:3300/0 conn(0x7fdfb013cc60 msgr2=0x7fdf9c00a100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:42:09.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.71:3300/0 conn(0x7fdfb013cc60 0x7fdf9c00a100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:09.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 1 -- 172.21.15.62:0/698029401 >> v2:172.21.15.62:3300/0 conn(0x7fdfb00a60b0 msgr2=0x7fdf9c00a660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:42:09.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 1 --2- 172.21.15.62:0/698029401 >> v2:172.21.15.62:3300/0 conn(0x7fdfb00a60b0 0x7fdf9c00a660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:09.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 10 monclient: _finish_hunting 0 2024-04-19T01:42:09.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 1 monclient: found mon.noname-c-legacy 2024-04-19T01:42:09.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:09.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.71:6789/0 2024-04-19T01:42:09.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 1 -- 172.21.15.62:0/698029401 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf9c00ace0 con 0x7fdf9c007800 2024-04-19T01:42:09.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 10 monclient: _finish_auth 0 2024-04-19T01:42:09.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 10 monclient: _check_auth_tickets 2024-04-19T01:42:09.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:09.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 1 -- 172.21.15.62:0/698029401 <== mon.1 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1244427781 0 0) 0x7fdfa8009c70 con 0x7fdf9c007800 2024-04-19T01:42:09.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:09.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-19T01:42:09.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 10 monclient: dump: 2024-04-19T01:42:09.917 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:09.917 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:09.917 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:09.918 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:09.918 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:09.918 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:09.918 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:09.918 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:09.918 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:09.918 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:09.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] but i'm connected to v1:172.21.15.71:6789/0, reconnecting 2024-04-19T01:42:09.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:09.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 1 -- 172.21.15.62:0/698029401 >> v1:172.21.15.71:6789/0 conn(0x7fdf9c007800 legacy=0x7fdf9c003b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:09.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-19T01:42:09.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 1 --2- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf98003a10 0x7fdf98003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:09.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 10 monclient(hunting): picked mon.a con 0x7fdf98003a10 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:09.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 1 --2- 172.21.15.62:0/698029401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf98004810 0x7fdf98004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:09.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 10 monclient(hunting): picked mon.b con 0x7fdf98004810 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:09.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 1 --2- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf98005220 0x7fdf9800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:09.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 10 monclient(hunting): picked mon.c con 0x7fdf98005220 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:09.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:09.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 1 -- 172.21.15.62:0/698029401 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb009bd20 con 0x7fdf98003a10 2024-04-19T01:42:09.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:09.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 1 -- 172.21.15.62:0/698029401 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb009c000 con 0x7fdf98004810 2024-04-19T01:42:09.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:09.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 1 -- 172.21.15.62:0/698029401 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fdf9800c1d0 con 0x7fdf98005220 2024-04-19T01:42:09.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:09.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaf7fe640 10 monclient(hunting): _finish_auth 0 2024-04-19T01:42:09.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfb568c640 1 --2- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf98005220 0x7fdf9800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:09.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfb4e8b640 1 --2- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf98003a10 0x7fdf98003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:09.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaffff640 1 --2- 172.21.15.62:0/698029401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf98004810 0x7fdf98004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:09.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfb4e8b640 10 monclient(hunting): get_auth_request con 0x7fdf98003a10 auth_method 0 2024-04-19T01:42:09.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfb4e8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:09.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfb4e8b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:09.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfb4e8b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:42:09.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfb568c640 10 monclient(hunting): get_auth_request con 0x7fdf98005220 auth_method 0 2024-04-19T01:42:09.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfb568c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:09.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfb568c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:09.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfb568c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:42:09.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaffff640 10 monclient(hunting): get_auth_request con 0x7fdf98004810 auth_method 0 2024-04-19T01:42:09.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:09.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:09.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:42:09.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfb4e8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:09.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfb4e8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:09.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfb4e8b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T01:42:09.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:09.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:09.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T01:42:09.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfb568c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:09.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfb568c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:09.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfb568c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T01:42:09.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfb4e8b640 10 monclient(hunting): handle_auth_done global_id 4099 payload 293 2024-04-19T01:42:09.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfb4e8b640 1 -- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf98005220 msgr2=0x7fdf9800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:09.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfb4e8b640 1 --2- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf98005220 0x7fdf9800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:09.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfb4e8b640 1 -- 172.21.15.62:0/698029401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf98004810 msgr2=0x7fdf98004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:09.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfb4e8b640 1 --2- 172.21.15.62:0/698029401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf98004810 0x7fdf98004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:09.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfb4e8b640 10 monclient: _finish_hunting 0 2024-04-19T01:42:09.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfb4e8b640 1 monclient: found mon.a 2024-04-19T01:42:09.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfb4e8b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:09.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfb4e8b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:09.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfb4e8b640 1 -- 172.21.15.62:0/698029401 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf98003530 con 0x7fdf98003a10 2024-04-19T01:42:09.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfb4e8b640 10 monclient: _finish_auth 0 2024-04-19T01:42:09.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfb4e8b640 10 monclient: _check_auth_tickets 2024-04-19T01:42:09.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfb4e8b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:09.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfb70ef640 5 monclient: authenticate success, global_id 4099 2024-04-19T01:42:09.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfb70ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:42:09.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfaffff640 1 --2- 172.21.15.62:0/698029401 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf98004810 0x7fdf98004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:42:09.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.914+0000 7fdfb568c640 1 -- 172.21.15.62:0/698029401 reap_dead start 2024-04-19T01:42:09.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.922+0000 7fdfb4e8b640 1 -- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf98003a10 msgr2=0x7fdf98003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:09.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.922+0000 7fdfb4e8b640 1 --2- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf98003a10 0x7fdf98003e30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c0072f0 tx=0x7fdfa001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa64376f2959e7b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:09.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.922+0000 7fdfaf7fe640 1 -- 172.21.15.62:0/698029401 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa00b0070 con 0x7fdf98003a10 2024-04-19T01:42:09.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.922+0000 7fdfaf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:09.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.922+0000 7fdfaf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:09.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.922+0000 7fdfaf7fe640 10 monclient: dump: 2024-04-19T01:42:09.924 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:09.924 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:09.924 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:09.924 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:09.924 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:09.924 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:09.924 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:09.924 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:09.924 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:09.924 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:09.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.922+0000 7fdfb70ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:42:09.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.922+0000 7fdfaf7fe640 1 -- 172.21.15.62:0/698029401 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfa00070d0 con 0x7fdf98003a10 2024-04-19T01:42:09.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.922+0000 7fdfaf7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:09.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.922+0000 7fdfb70ef640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:09.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.922+0000 7fdfb70ef640 4 set_mon_vals no callback set 2024-04-19T01:42:09.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.922+0000 7fdfaf7fe640 1 -- 172.21.15.62:0/698029401 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa0001640 con 0x7fdf98003a10 2024-04-19T01:42:09.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.922+0000 7fdfaf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:09.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.922+0000 7fdfaf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:09.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.922+0000 7fdfaf7fe640 10 monclient: dump: 2024-04-19T01:42:09.925 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:09.925 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:09.925 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:09.925 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:09.925 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:09.925 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:09.925 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:09.926 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:09.926 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:09.926 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:09.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.922+0000 7fdfb70ef640 10 monclient: shutdown 2024-04-19T01:42:09.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.922+0000 7fdfb70ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:09.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.922+0000 7fdfb70ef640 1 -- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf98003a10 msgr2=0x7fdf98003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:09.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.922+0000 7fdfb70ef640 1 --2- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf98003a10 0x7fdf98003e30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c0072f0 tx=0x7fdfa001c520 comp rx=0 tx=0).stop 2024-04-19T01:42:09.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.922+0000 7fdfb70ef640 1 -- 172.21.15.62:0/698029401 shutdown_connections 2024-04-19T01:42:09.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.922+0000 7fdfb70ef640 1 --2- 172.21.15.62:0/698029401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf98003a10 0x7fdf98003e30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:09.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.922+0000 7fdfb70ef640 1 -- 172.21.15.62:0/698029401 >> 172.21.15.62:0/698029401 conn(0x7fdfb00aa1d0 msgr2=0x7fdfb00a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:09.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.922+0000 7fdfb70ef640 1 -- 172.21.15.62:0/698029401 shutdown_connections 2024-04-19T01:42:09.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.922+0000 7fdfb70ef640 1 -- 172.21.15.62:0/698029401 wait complete. 2024-04-19T01:42:09.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb70ef640 10 monclient: build_initial_monmap 2024-04-19T01:42:09.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb70ef640 10 monclient: monmap: 2024-04-19T01:42:09.928 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:09.928 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:09.928 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:09.928 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:09.928 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:09.928 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:09.928 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:09.928 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:09.928 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:09.928 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:09.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb70ef640 1 Processor -- start 2024-04-19T01:42:09.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb70ef640 1 -- start start 2024-04-19T01:42:09.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb70ef640 10 monclient: init 2024-04-19T01:42:09.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb70ef640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:09.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb70ef640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-19T01:42:09.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb70ef640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf98003a10 0x7fdfb01d02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:09.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb70ef640 10 monclient(hunting): picked mon.noname-c con 0x7fdf98003a10 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:09.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb70ef640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb01d07e0 0x7fdfb01208b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:09.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb70ef640 10 monclient(hunting): picked mon.noname-a con 0x7fdfb01d07e0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:09.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb70ef640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdfb01d0c00 0x7fdfb0120df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:09.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb70ef640 10 monclient(hunting): picked mon.noname-b con 0x7fdfb01d0c00 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:09.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb70ef640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:09.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb70ef640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb00a4f60 con 0x7fdfb01d07e0 2024-04-19T01:42:09.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb70ef640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:09.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb70ef640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb00a4de0 con 0x7fdfb01d0c00 2024-04-19T01:42:09.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb70ef640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:09.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb70ef640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fdfb00a50e0 con 0x7fdf98003a10 2024-04-19T01:42:09.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb70ef640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:09.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb70ef640 10 monclient(hunting): authenticate will time out at 915.750653s 2024-04-19T01:42:09.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfaffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb01d07e0 0x7fdfb01208b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:09.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfaffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb01d07e0 0x7fdfb01208b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52780/0 (socket says 172.21.15.62:52780) 2024-04-19T01:42:09.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfaffff640 1 -- 172.21.15.62:0/1907304419 learned_addr learned my addr 172.21.15.62:0/1907304419 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:09.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfaffff640 10 monclient(hunting): get_auth_request con 0x7fdfb01d07e0 auth_method 0 2024-04-19T01:42:09.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb4e8b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf98003a10 0x7fdfb01d02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:09.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:09.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfaffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:09.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfaffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:09.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb4e8b640 10 monclient(hunting): get_auth_request con 0x7fdf98003a10 auth_method 0 2024-04-19T01:42:09.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb4e8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:09.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb4e8b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:09.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb4e8b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:09.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfaffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:09.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfaffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:09.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb568c640 1 --2- 172.21.15.62:0/1907304419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdfb01d0c00 0x7fdfb0120df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:09.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfaffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:09.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb4e8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:09.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb4e8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:09.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb4e8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:09.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb568c640 10 monclient(hunting): get_auth_request con 0x7fdfb01d0c00 auth_method 0 2024-04-19T01:42:09.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb568c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:09.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb568c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:09.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb568c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:09.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb568c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:09.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb568c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:09.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb568c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:09.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfaffff640 10 monclient(hunting): handle_auth_done global_id 4098 payload 824 2024-04-19T01:42:09.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfaffff640 1 -- 172.21.15.62:0/1907304419 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf98003a10 msgr2=0x7fdfb01d02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:09.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfaffff640 1 --2- 172.21.15.62:0/1907304419 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf98003a10 0x7fdfb01d02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:09.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfaffff640 1 -- 172.21.15.62:0/1907304419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdfb01d0c00 msgr2=0x7fdfb0120df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:09.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb568c640 1 -- 172.21.15.62:0/1907304419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdfb01d0c00 msgr2=0x7fdfb0120df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-19T01:42:09.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb568c640 1 -- 172.21.15.62:0/1907304419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdfb01d0c00 msgr2=0x7fdfb0120df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:42:09.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb568c640 1 --2- 172.21.15.62:0/1907304419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdfb01d0c00 0x7fdfb0120df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:42:09.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb568c640 1 --2- 172.21.15.62:0/1907304419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdfb01d0c00 0x7fdfb0120df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:42:09.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfaffff640 1 --2- 172.21.15.62:0/1907304419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdfb01d0c00 0x7fdfb0120df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:09.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfaffff640 10 monclient: _finish_hunting 0 2024-04-19T01:42:09.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfaffff640 1 monclient: found mon.noname-a 2024-04-19T01:42:09.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfaffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:09.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfaffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:09.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfaffff640 1 -- 172.21.15.62:0/1907304419 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfb0122400 con 0x7fdfb01d07e0 2024-04-19T01:42:09.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.926+0000 7fdfb4e8b640 1 --2- 172.21.15.62:0/1907304419 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf98003a10 0x7fdfb01d02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:42:09.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.930+0000 7fdfaffff640 1 -- 172.21.15.62:0/1907304419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb01d07e0 msgr2=0x7fdfb01208b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:09.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.930+0000 7fdfaffff640 1 --2- 172.21.15.62:0/1907304419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb01d07e0 0x7fdfb01208b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4004660 tx=0x7fdfa4016ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:09.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.930+0000 7fdfad7fa640 1 -- 172.21.15.62:0/1907304419 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa400f040 con 0x7fdfb01d07e0 2024-04-19T01:42:09.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.930+0000 7fdfad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:09.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.930+0000 7fdfad7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:09.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.930+0000 7fdfad7fa640 10 monclient: dump: 2024-04-19T01:42:09.938 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:09.938 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:09.938 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:09.938 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:09.938 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:09.938 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:09.938 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:09.938 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:09.938 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:09.938 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:09.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.930+0000 7fdfad7fa640 10 monclient: _finish_auth 0 2024-04-19T01:42:09.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.930+0000 7fdfad7fa640 10 monclient: _check_auth_tickets 2024-04-19T01:42:09.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.930+0000 7fdfad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:09.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.930+0000 7fdfad7fa640 1 -- 172.21.15.62:0/1907304419 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfa4029070 con 0x7fdfb01d07e0 2024-04-19T01:42:09.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.930+0000 7fdfad7fa640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:09.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.930+0000 7fdfad7fa640 1 -- 172.21.15.62:0/1907304419 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa4018450 con 0x7fdfb01d07e0 2024-04-19T01:42:09.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.930+0000 7fdfad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:09.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.930+0000 7fdfad7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:09.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.930+0000 7fdfad7fa640 10 monclient: dump: 2024-04-19T01:42:09.939 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:09.939 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:09.939 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:09.939 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:09.939 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:09.939 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:09.939 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:09.940 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:09.940 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:09.940 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:09.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.930+0000 7fdfb70ef640 5 monclient: authenticate success, global_id 4098 2024-04-19T01:42:09.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.930+0000 7fdfb70ef640 10 monclient: _renew_subs 2024-04-19T01:42:09.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.930+0000 7fdfb70ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:09.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.930+0000 7fdfb70ef640 1 -- 172.21.15.62:0/1907304419 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfb01246f0 con 0x7fdfb01d07e0 2024-04-19T01:42:09.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.930+0000 7fdfaeffd640 4 set_mon_vals no callback set 2024-04-19T01:42:09.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.930+0000 7fdfb70ef640 10 monclient: _renew_subs 2024-04-19T01:42:09.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.930+0000 7fdfb70ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:09.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.930+0000 7fdfb70ef640 1 -- 172.21.15.62:0/1907304419 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfb0124410 con 0x7fdfb01d07e0 2024-04-19T01:42:09.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.930+0000 7fdfad7fa640 1 -- 172.21.15.62:0/1907304419 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdfa4023070 con 0x7fdfb01d07e0 2024-04-19T01:42:09.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.930+0000 7fdfad7fa640 1 -- 172.21.15.62:0/1907304419 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fdfa402e550 con 0x7fdfb01d07e0 2024-04-19T01:42:09.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.930+0000 7fdfb70ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:09.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.930+0000 7fdfb70ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:09.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.930+0000 7fdfb70ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:09.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.930+0000 7fdfb70ef640 1 -- 172.21.15.62:0/1907304419 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfb0068ab0 con 0x7fdfb01d07e0 2024-04-19T01:42:09.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.934+0000 7fdfad7fa640 1 -- 172.21.15.62:0/1907304419 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fdfa4021040 con 0x7fdfb01d07e0 2024-04-19T01:42:09.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.934+0000 7fdfad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:09.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.934+0000 7fdfad7fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:10.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.998+0000 7fdfb70ef640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-04-19T01:42:10.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.998+0000 7fdfb70ef640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-04-19T01:42:10.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.998+0000 7fdfb70ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:10.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:09.998+0000 7fdfb70ef640 1 -- 172.21.15.62:0/1907304419 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fdfb00a3200 con 0x7fdfb01d07e0 2024-04-19T01:42:10.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:10.002+0000 7fdfad7fa640 1 -- 172.21.15.62:0/1907304419 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fdfa40059a0 con 0x7fdfb01d07e0 2024-04-19T01:42:10.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:10.002+0000 7fdfad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-04-19T01:42:10.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:10.002+0000 7fdfad7fa640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-04-19T01:42:10.003 INFO:teuthology.orchestra.run.smithi062.stderr:adjusted tunables profile to default 2024-04-19T01:42:10.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:10.002+0000 7fdfb70ef640 10 monclient: shutdown 2024-04-19T01:42:10.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:10.002+0000 7fdfb70ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:10.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:10.002+0000 7fdfb70ef640 1 -- 172.21.15.62:0/1907304419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb01d07e0 msgr2=0x7fdfb01208b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:10.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:10.002+0000 7fdfb70ef640 1 --2- 172.21.15.62:0/1907304419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb01d07e0 0x7fdfb01208b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4004660 tx=0x7fdfa4016ee0 comp rx=0 tx=0).stop 2024-04-19T01:42:10.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:10.002+0000 7fdfb70ef640 1 -- 172.21.15.62:0/1907304419 shutdown_connections 2024-04-19T01:42:10.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:10.002+0000 7fdfb70ef640 1 --2- 172.21.15.62:0/1907304419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdfb01d0c00 0x7fdfb0120df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:10.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:10.002+0000 7fdfb70ef640 1 --2- 172.21.15.62:0/1907304419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfb01d07e0 0x7fdfb01208b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:10.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:10.002+0000 7fdfb70ef640 1 --2- 172.21.15.62:0/1907304419 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdf98003a10 0x7fdfb01d02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:10.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:10.002+0000 7fdfb70ef640 1 -- 172.21.15.62:0/1907304419 >> 172.21.15.62:0/1907304419 conn(0x7fdfb00aa1d0 msgr2=0x7fdfb013bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:10.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:10.002+0000 7fdfb70ef640 1 -- 172.21.15.62:0/1907304419 shutdown_connections 2024-04-19T01:42:10.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:10.002+0000 7fdfb70ef640 1 -- 172.21.15.62:0/1907304419 wait complete. 2024-04-19T01:42:10.023 INFO:tasks.ceph:check_enable_crimson: False 2024-04-19T01:42:10.023 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-19T01:42:10.023 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-19T01:42:10.024 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-19T01:42:10.035 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-19T01:42:10.035 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-19T01:42:10.090 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-19T01:42:10.090 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-19T01:42:10.147 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-19T01:42:10.147 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-19T01:42:10.161 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-19T01:42:10.161 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-19T01:42:10.215 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-19T01:42:10.215 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-19T01:42:10.275 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd new 64304e60-9fe6-413b-8559-3b8ebd179a7f 0 2024-04-19T01:42:10.325 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-04-19T01:42:10.329+0000 7f76bc1b9180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-19T01:42:10.441 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.445+0000 7fb542eea640 10 monclient: get_monmap_and_config 2024-04-19T01:42:10.441 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.445+0000 7fb542eea640 10 monclient: build_initial_monmap 2024-04-19T01:42:10.441 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.445+0000 7fb542eea640 10 monclient: monmap: 2024-04-19T01:42:10.441 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-04-19T01:42:10.441 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:10.441 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-04-19T01:42:10.447293+0000 2024-04-19T01:42:10.441 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-04-19T01:42:10.447293+0000 2024-04-19T01:42:10.442 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:10.442 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:10.442 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:10.442 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:10.442 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:10.442 INFO:teuthology.orchestra.run.smithi071.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:10.442 INFO:teuthology.orchestra.run.smithi071.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:10.442 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:10.444 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.445+0000 7fb542eea640 1 Processor -- start 2024-04-19T01:42:10.444 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.445+0000 7fb542eea640 1 -- start start 2024-04-19T01:42:10.444 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.445+0000 7fb542eea640 10 monclient: init 2024-04-19T01:42:10.444 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.445+0000 7fb542eea640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:10.444 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.445+0000 7fb542eea640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-04-19T01:42:10.444 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.445+0000 7fb542eea640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb53c0a3bd0 addr v1:172.21.15.71:6789/0 2024-04-19T01:42:10.444 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.445+0000 7fb542eea640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb53c0a2f30 0x7fb53c13c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:10.445 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.445+0000 7fb542eea640 10 monclient(hunting): picked mon.noname-b con 0x7fb53c0a2f30 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:10.445 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.445+0000 7fb542eea640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb53c13cac0 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:10.445 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.445+0000 7fb542eea640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb53c09fe80 con 0x7fb53c13cac0 2024-04-19T01:42:10.445 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.445+0000 7fb542eea640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb53c0a0ae0 con 0x7fb53c0a3bd0 2024-04-19T01:42:10.445 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.445+0000 7fb542eea640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:10.445 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.445+0000 7fb542eea640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fb53c0a0120 con 0x7fb53c0a2f30 2024-04-19T01:42:10.445 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.445+0000 7fb542eea640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:10.445 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.445+0000 7fb542eea640 10 monclient(hunting): authenticate will time out at 915.911775s 2024-04-19T01:42:10.445 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.445+0000 7fb540c86640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fb53c0a3bd0 0x7fb53c0a2820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:47922/0 (socket says 172.21.15.71:47922) 2024-04-19T01:42:10.445 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.445+0000 7fb540c86640 1 -- 172.21.15.71:0/29453559 learned_addr learned my addr 172.21.15.71:0/29453559 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-04-19T01:42:10.445 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.449+0000 7fb533fff640 1 --2- 172.21.15.71:0/29453559 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb53c0a2f30 0x7fb53c13c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:10.445 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.449+0000 7fb533fff640 10 monclient(hunting): get_auth_request con 0x7fb53c0a2f30 auth_method 0 2024-04-19T01:42:10.445 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.449+0000 7fb533fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:10.445 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.449+0000 7fb533fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:10.445 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.449+0000 7fb533fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:10.446 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.449+0000 7fb533fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:10.446 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.449+0000 7fb533fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:10.446 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.449+0000 7fb533fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:10.446 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.449+0000 7fb533fff640 10 monclient(hunting): handle_auth_done global_id 4109 payload 274 2024-04-19T01:42:10.446 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.449+0000 7fb533fff640 1 -- 172.21.15.71:0/29453559 >> v1:172.21.15.71:6789/0 conn(0x7fb53c0a3bd0 legacy=0x7fb53c0a2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:10.446 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.449+0000 7fb533fff640 1 -- 172.21.15.71:0/29453559 >> v1:172.21.15.62:6789/0 conn(0x7fb53c13cac0 legacy=0x7fb53c13eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:10.446 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.449+0000 7fb533fff640 10 monclient: _finish_hunting 0 2024-04-19T01:42:10.446 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.449+0000 7fb533fff640 1 monclient: found mon.noname-b 2024-04-19T01:42:10.446 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.449+0000 7fb533fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:10.446 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.449+0000 7fb533fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:42:10.446 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.449+0000 7fb533fff640 1 -- 172.21.15.71:0/29453559 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb53c0a0dc0 con 0x7fb53c0a2f30 2024-04-19T01:42:10.452 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.453+0000 7fb533fff640 1 -- 172.21.15.71:0/29453559 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb53c0a2f30 msgr2=0x7fb53c13c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:10.452 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.453+0000 7fb533fff640 1 --2- 172.21.15.71:0/29453559 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb53c0a2f30 0x7fb53c13c580 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb53c0762d0 tx=0x7fb52c01c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2439ad307c4c89fa server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:10.453 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.453+0000 7fb533fff640 1 -- 172.21.15.71:0/29453559 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb53c0a2f30 msgr2=0x7fb53c13c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-19T01:42:10.453 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.453+0000 7fb533fff640 1 -- 172.21.15.71:0/29453559 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb53c0a2f30 msgr2=0x7fb53c13c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:42:10.453 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.453+0000 7fb533fff640 1 --2- 172.21.15.71:0/29453559 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb53c0a2f30 0x7fb53c13c580 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb53c0762d0 tx=0x7fb52c01c380 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:42:10.453 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.453+0000 7fb533fff640 1 --2- 172.21.15.71:0/29453559 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb53c0a2f30 0x7fb53c13c580 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb53c0762d0 tx=0x7fb52c01c380 comp rx=0 tx=0).stop 2024-04-19T01:42:10.453 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.453+0000 7fb5337fe640 10 monclient: ms_handle_reset current mon [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:10.453 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.453+0000 7fb5337fe640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:10.453 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.453+0000 7fb5337fe640 1 -- 172.21.15.71:0/29453559 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb53c0a2f30 msgr2=0x7fb53c13c580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:42:10.453 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.453+0000 7fb5337fe640 1 --2- 172.21.15.71:0/29453559 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb53c0a2f30 0x7fb53c13c580 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:10.453 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.453+0000 7fb5337fe640 10 monclient: _add_conns ranks=[3,4] 2024-04-19T01:42:10.453 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.453+0000 7fb5337fe640 1 --2- 172.21.15.71:0/29453559 >> v2:172.21.15.62:3300/0 conn(0x7fb524003510 0x7fb524005990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:10.453 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.453+0000 7fb5337fe640 10 monclient(hunting): picked mon.noname-a con 0x7fb524003510 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:10.453 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.453+0000 7fb5337fe640 1 --2- 172.21.15.71:0/29453559 >> v2:172.21.15.71:3300/0 conn(0x7fb5240062c0 0x7fb524008780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:10.453 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.453+0000 7fb5337fe640 10 monclient(hunting): picked mon.noname-c con 0x7fb5240062c0 addr v2:172.21.15.71:3300/0 2024-04-19T01:42:10.453 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.453+0000 7fb5337fe640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:10.453 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.453+0000 7fb5337fe640 1 -- 172.21.15.71:0/29453559 --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fb524009050 con 0x7fb524003510 2024-04-19T01:42:10.454 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.453+0000 7fb5337fe640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:10.454 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.453+0000 7fb5337fe640 1 -- 172.21.15.71:0/29453559 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fb524009200 con 0x7fb5240062c0 2024-04-19T01:42:10.454 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.453+0000 7fb5337fe640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:10.454 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.453+0000 7fb5337fe640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:10.454 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.453+0000 7fb533fff640 1 --2- 172.21.15.71:0/29453559 >> v2:172.21.15.71:3300/0 conn(0x7fb5240062c0 0x7fb524008780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:10.454 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.453+0000 7fb533fff640 10 monclient(hunting): get_auth_request con 0x7fb5240062c0 auth_method 0 2024-04-19T01:42:10.454 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.453+0000 7fb533fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:10.454 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.453+0000 7fb533fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:10.454 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.453+0000 7fb533fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:42:10.454 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.453+0000 7fb540c86640 1 --2- 172.21.15.71:0/29453559 >> v2:172.21.15.62:3300/0 conn(0x7fb524003510 0x7fb524005990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:10.454 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.453+0000 7fb533fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:10.454 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.453+0000 7fb533fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:10.454 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb533fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T01:42:10.454 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb540c86640 10 monclient(hunting): get_auth_request con 0x7fb524003510 auth_method 0 2024-04-19T01:42:10.455 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb540c86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:10.455 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb540c86640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:10.455 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb540c86640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:42:10.455 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb540c86640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:10.455 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb540c86640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:10.455 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb540c86640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T01:42:10.455 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb533fff640 10 monclient(hunting): handle_auth_done global_id 4109 payload 293 2024-04-19T01:42:10.455 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb533fff640 1 -- 172.21.15.71:0/29453559 >> v2:172.21.15.62:3300/0 conn(0x7fb524003510 msgr2=0x7fb524005990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:10.455 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb533fff640 1 --2- 172.21.15.71:0/29453559 >> v2:172.21.15.62:3300/0 conn(0x7fb524003510 0x7fb524005990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:10.455 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb533fff640 10 monclient: _finish_hunting 0 2024-04-19T01:42:10.455 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb533fff640 1 monclient: found mon.noname-c 2024-04-19T01:42:10.455 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb533fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:10.455 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb533fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-04-19T01:42:10.455 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb533fff640 1 -- 172.21.15.71:0/29453559 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb524009470 con 0x7fb5240062c0 2024-04-19T01:42:10.460 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb533fff640 1 -- 172.21.15.71:0/29453559 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb5240062c0 msgr2=0x7fb524008780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:10.460 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb533fff640 1 --2- 172.21.15.71:0/29453559 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb5240062c0 0x7fb524008780 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb53c06ccb0 tx=0x7fb52c009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f2fa6f9663a5f73 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:10.460 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb5337fe640 1 -- 172.21.15.71:0/29453559 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb52c0b0070 con 0x7fb5240062c0 2024-04-19T01:42:10.460 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb5337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:10.460 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb5337fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:10.460 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb5337fe640 10 monclient: dump: 2024-04-19T01:42:10.460 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-04-19T01:42:10.461 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:10.461 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:10.461 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:10.461 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:10.461 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:10.461 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:10.461 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:10.461 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:10.461 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:10.461 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb5337fe640 10 monclient: _finish_auth 0 2024-04-19T01:42:10.461 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb5337fe640 10 monclient: _check_auth_tickets 2024-04-19T01:42:10.461 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb5337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:10.461 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb5337fe640 1 -- 172.21.15.71:0/29453559 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb52c003820 con 0x7fb5240062c0 2024-04-19T01:42:10.461 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb542eea640 5 monclient: authenticate success, global_id 4109 2024-04-19T01:42:10.461 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb542eea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:42:10.462 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb5337fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:10.462 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb5337fe640 1 -- 172.21.15.71:0/29453559 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb52c003bd0 con 0x7fb5240062c0 2024-04-19T01:42:10.462 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb542eea640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:10.462 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb542eea640 4 set_mon_vals no callback set 2024-04-19T01:42:10.462 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb5337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:10.462 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb5337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:42:10.462 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb5337fe640 10 monclient: dump: 2024-04-19T01:42:10.462 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-04-19T01:42:10.462 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:10.462 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:10.462 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:10.462 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:10.462 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:10.462 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:10.462 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:10.463 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:10.463 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:10.463 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb542eea640 10 monclient: shutdown 2024-04-19T01:42:10.463 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb542eea640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:10.463 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb542eea640 1 -- 172.21.15.71:0/29453559 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb5240062c0 msgr2=0x7fb524008780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:10.463 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb542eea640 1 --2- 172.21.15.71:0/29453559 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb5240062c0 0x7fb524008780 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb53c06ccb0 tx=0x7fb52c009190 comp rx=0 tx=0).stop 2024-04-19T01:42:10.463 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb541487640 1 -- 172.21.15.71:0/29453559 reap_dead start 2024-04-19T01:42:10.463 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb542eea640 1 -- 172.21.15.71:0/29453559 shutdown_connections 2024-04-19T01:42:10.463 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb542eea640 1 --2- 172.21.15.71:0/29453559 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb5240062c0 0x7fb524008780 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:10.463 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb542eea640 1 -- 172.21.15.71:0/29453559 >> 172.21.15.71:0/29453559 conn(0x7fb53c0a7070 msgr2=0x7fb53c0a3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:10.463 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb542eea640 1 -- 172.21.15.71:0/29453559 shutdown_connections 2024-04-19T01:42:10.463 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.457+0000 7fb542eea640 1 -- 172.21.15.71:0/29453559 wait complete. 2024-04-19T01:42:10.463 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb542eea640 10 monclient: build_initial_monmap 2024-04-19T01:42:10.463 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb542eea640 10 monclient: monmap: 2024-04-19T01:42:10.463 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-04-19T01:42:10.464 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:10.464 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 0.000000 2024-04-19T01:42:10.464 INFO:teuthology.orchestra.run.smithi071.stderr:created 0.000000 2024-04-19T01:42:10.464 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:10.464 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:10.464 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:10.464 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:10.464 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:10.464 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:10.464 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb542eea640 1 Processor -- start 2024-04-19T01:42:10.464 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb542eea640 1 -- start start 2024-04-19T01:42:10.464 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb542eea640 10 monclient: init 2024-04-19T01:42:10.464 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb542eea640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:10.465 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb542eea640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-19T01:42:10.465 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb542eea640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb5240062c0 0x7fb53c13b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:10.465 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb542eea640 10 monclient(hunting): picked mon.noname-c con 0x7fb5240062c0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:10.465 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb542eea640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb53c11cfa0 0x7fb53c136190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:10.465 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb542eea640 10 monclient(hunting): picked mon.noname-b con 0x7fb53c11cfa0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:10.465 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb542eea640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb53c1cf430 0x7fb53c1cf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:10.465 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb542eea640 10 monclient(hunting): picked mon.noname-a con 0x7fb53c1cf430 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:10.465 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb542eea640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:10.465 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb542eea640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb53c143000 con 0x7fb53c1cf430 2024-04-19T01:42:10.465 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb542eea640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:10.465 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb542eea640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb53c0a0560 con 0x7fb53c11cfa0 2024-04-19T01:42:10.465 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb542eea640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:10.466 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb542eea640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fb53c143180 con 0x7fb5240062c0 2024-04-19T01:42:10.466 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb542eea640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:10.466 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb542eea640 10 monclient(hunting): authenticate will time out at 915.925381s 2024-04-19T01:42:10.466 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb533fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb53c11cfa0 0x7fb53c136190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:10.466 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb533fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb53c11cfa0 0x7fb53c136190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52100/0 (socket says 172.21.15.71:52100) 2024-04-19T01:42:10.466 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb533fff640 1 -- 172.21.15.71:0/295359582 learned_addr learned my addr 172.21.15.71:0/295359582 (peer_addr_for_me v2:172.21.15.71:0/0) 2024-04-19T01:42:10.466 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb533fff640 10 monclient(hunting): get_auth_request con 0x7fb53c11cfa0 auth_method 0 2024-04-19T01:42:10.466 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb533fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:10.466 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb533fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:10.466 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb533fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:10.467 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb533fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:10.467 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb533fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:10.467 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb533fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:10.467 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb541487640 1 --2- 172.21.15.71:0/295359582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb53c1cf430 0x7fb53c1cf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:10.467 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb540c86640 1 --2- 172.21.15.71:0/295359582 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb5240062c0 0x7fb53c13b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:10.467 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb541487640 10 monclient(hunting): get_auth_request con 0x7fb53c1cf430 auth_method 0 2024-04-19T01:42:10.467 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb541487640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:10.467 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb541487640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:10.467 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb541487640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:10.467 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb540c86640 10 monclient(hunting): get_auth_request con 0x7fb5240062c0 auth_method 0 2024-04-19T01:42:10.467 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb540c86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:10.468 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb540c86640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:10.468 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb540c86640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:10.468 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb533fff640 10 monclient(hunting): handle_auth_done global_id 4111 payload 824 2024-04-19T01:42:10.468 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb533fff640 1 -- 172.21.15.71:0/295359582 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb5240062c0 msgr2=0x7fb53c13b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:10.468 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb533fff640 1 --2- 172.21.15.71:0/295359582 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb5240062c0 0x7fb53c13b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:10.468 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb533fff640 1 -- 172.21.15.71:0/295359582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb53c1cf430 msgr2=0x7fb53c1cf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:10.468 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb533fff640 1 --2- 172.21.15.71:0/295359582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb53c1cf430 0x7fb53c1cf8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:10.468 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb533fff640 10 monclient: _finish_hunting 0 2024-04-19T01:42:10.468 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb533fff640 1 monclient: found mon.noname-b 2024-04-19T01:42:10.468 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb533fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:10.468 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb533fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.71:3300/0 2024-04-19T01:42:10.469 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.461+0000 7fb533fff640 1 -- 172.21.15.71:0/295359582 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb53c11df60 con 0x7fb53c11cfa0 2024-04-19T01:42:10.469 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.465+0000 7fb533fff640 1 -- 172.21.15.71:0/295359582 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb53c11cfa0 msgr2=0x7fb53c136190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:10.469 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.465+0000 7fb533fff640 1 --2- 172.21.15.71:0/295359582 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb53c11cfa0 0x7fb53c136190 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb52c097b70 tx=0x7fb52c019120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:10.469 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.465+0000 7fb5317fa640 1 -- 172.21.15.71:0/295359582 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb52c0b0040 con 0x7fb53c11cfa0 2024-04-19T01:42:10.469 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.465+0000 7fb5317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:10.469 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.465+0000 7fb5317fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:42:10.469 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.465+0000 7fb5317fa640 10 monclient: dump: 2024-04-19T01:42:10.469 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-04-19T01:42:10.469 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:10.469 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:10.469 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:10.470 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:10.470 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:10.470 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:10.470 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:10.470 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:10.470 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:10.470 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.465+0000 7fb5317fa640 10 monclient: _finish_auth 0 2024-04-19T01:42:10.470 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.465+0000 7fb5317fa640 10 monclient: _check_auth_tickets 2024-04-19T01:42:10.470 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.465+0000 7fb5317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:10.470 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.465+0000 7fb5317fa640 1 -- 172.21.15.71:0/295359582 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb52c004170 con 0x7fb53c11cfa0 2024-04-19T01:42:10.470 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.465+0000 7fb542eea640 5 monclient: authenticate success, global_id 4111 2024-04-19T01:42:10.470 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.465+0000 7fb542eea640 10 monclient: _renew_subs 2024-04-19T01:42:10.471 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.465+0000 7fb542eea640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:42:10.471 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.465+0000 7fb542eea640 1 -- 172.21.15.71:0/295359582 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb53c11f550 con 0x7fb53c11cfa0 2024-04-19T01:42:10.471 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.465+0000 7fb5317fa640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:10.471 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.465+0000 7fb5317fa640 1 -- 172.21.15.71:0/295359582 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb52c003e20 con 0x7fb53c11cfa0 2024-04-19T01:42:10.471 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.465+0000 7fb5317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:10.471 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.465+0000 7fb5317fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:42:10.471 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.465+0000 7fb5317fa640 10 monclient: dump: 2024-04-19T01:42:10.471 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-04-19T01:42:10.471 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:10.471 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:10.471 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:10.471 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:10.471 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:10.471 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:10.471 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:10.472 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:10.472 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:10.472 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.465+0000 7fb532ffd640 4 set_mon_vals no callback set 2024-04-19T01:42:10.472 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.465+0000 7fb542eea640 10 monclient: _renew_subs 2024-04-19T01:42:10.472 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.465+0000 7fb542eea640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:42:10.472 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.465+0000 7fb542eea640 1 -- 172.21.15.71:0/295359582 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb53c072b40 con 0x7fb53c11cfa0 2024-04-19T01:42:10.472 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.465+0000 7fb5317fa640 1 -- 172.21.15.71:0/295359582 <== mon.1 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb52c0c9470 con 0x7fb53c11cfa0 2024-04-19T01:42:10.472 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.465+0000 7fb5317fa640 1 -- 172.21.15.71:0/295359582 <== mon.1 v2:172.21.15.71:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7fb52c0c05d0 con 0x7fb53c11cfa0 2024-04-19T01:42:10.472 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.465+0000 7fb542eea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:10.473 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.465+0000 7fb542eea640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:10.473 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.465+0000 7fb542eea640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:42:10.473 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.465+0000 7fb542eea640 1 -- 172.21.15.71:0/295359582 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb53c0a2050 con 0x7fb53c11cfa0 2024-04-19T01:42:10.473 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.473+0000 7fb5317fa640 1 -- 172.21.15.71:0/295359582 <== mon.1 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fb53c11df60 con 0x7fb53c11cfa0 2024-04-19T01:42:10.473 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.473+0000 7fb5317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:10.473 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.473+0000 7fb5317fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:10.536 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.541+0000 7fb542eea640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "64304e60-9fe6-413b-8559-3b8ebd179a7f", "id": 0}] 2024-04-19T01:42:10.536 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.541+0000 7fb542eea640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "64304e60-9fe6-413b-8559-3b8ebd179a7f", "id": 0}] 2024-04-19T01:42:10.536 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.541+0000 7fb542eea640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:42:10.536 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.541+0000 7fb542eea640 1 -- 172.21.15.71:0/295359582 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd new", "uuid": "64304e60-9fe6-413b-8559-3b8ebd179a7f", "id": 0} v 0) -- 0x7fb53c068ab0 con 0x7fb53c11cfa0 2024-04-19T01:42:10.541 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.545+0000 7fb5317fa640 1 -- 172.21.15.71:0/295359582 <== mon.1 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "64304e60-9fe6-413b-8559-3b8ebd179a7f", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7fb52c0ba030 con 0x7fb53c11cfa0 2024-04-19T01:42:10.541 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.545+0000 7fb5317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "64304e60-9fe6-413b-8559-3b8ebd179a7f", "id": 0}] 2024-04-19T01:42:10.542 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.545+0000 7fb5317fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:10.542 INFO:teuthology.orchestra.run.smithi071.stdout:0 2024-04-19T01:42:10.543 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.545+0000 7fb542eea640 10 monclient: shutdown 2024-04-19T01:42:10.543 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.545+0000 7fb542eea640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:10.543 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.545+0000 7fb542eea640 1 -- 172.21.15.71:0/295359582 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb53c11cfa0 msgr2=0x7fb53c136190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:10.543 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.545+0000 7fb542eea640 1 --2- 172.21.15.71:0/295359582 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb53c11cfa0 0x7fb53c136190 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb52c097b70 tx=0x7fb52c019120 comp rx=0 tx=0).stop 2024-04-19T01:42:10.543 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.545+0000 7fb542eea640 1 -- 172.21.15.71:0/295359582 shutdown_connections 2024-04-19T01:42:10.543 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.545+0000 7fb542eea640 1 --2- 172.21.15.71:0/295359582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb53c1cf430 0x7fb53c1cf8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:10.543 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.545+0000 7fb542eea640 1 --2- 172.21.15.71:0/295359582 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb53c11cfa0 0x7fb53c136190 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:10.543 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.545+0000 7fb542eea640 1 --2- 172.21.15.71:0/295359582 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb5240062c0 0x7fb53c13b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:10.543 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.545+0000 7fb542eea640 1 -- 172.21.15.71:0/295359582 >> 172.21.15.71:0/295359582 conn(0x7fb53c0a7070 msgr2=0x7fb53c0a1350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:10.543 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.545+0000 7fb542eea640 1 -- 172.21.15.71:0/295359582 shutdown_connections 2024-04-19T01:42:10.543 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.545+0000 7fb542eea640 1 -- 172.21.15.71:0/295359582 wait complete. 2024-04-19T01:42:10.559 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd new dec72cc0-fa6d-4fe9-b9cd-19dec591ab68 1 2024-04-19T01:42:10.724 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fee00462640 10 monclient: get_monmap_and_config 2024-04-19T01:42:10.725 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fee00462640 10 monclient: build_initial_monmap 2024-04-19T01:42:10.725 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fee00462640 10 monclient: monmap: 2024-04-19T01:42:10.725 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-04-19T01:42:10.725 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:10.726 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-04-19T01:42:10.730647+0000 2024-04-19T01:42:10.726 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-04-19T01:42:10.730647+0000 2024-04-19T01:42:10.726 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:10.726 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:10.726 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:10.726 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:10.726 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:10.726 INFO:teuthology.orchestra.run.smithi071.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:10.726 INFO:teuthology.orchestra.run.smithi071.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:10.726 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:10.728 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fee00462640 1 Processor -- start 2024-04-19T01:42:10.728 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fee00462640 1 -- start start 2024-04-19T01:42:10.729 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fee00462640 10 monclient: init 2024-04-19T01:42:10.729 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fee00462640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:10.729 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fee00462640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-04-19T01:42:10.729 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fee00462640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fedf81355e0 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:10.729 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fee00462640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fedf80726e0 0x7fedf8072b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:10.729 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fee00462640 10 monclient(hunting): picked mon.noname-a con 0x7fedf80726e0 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:10.729 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fee00462640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fedf813a480 0x7fedf813c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:10.729 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fee00462640 10 monclient(hunting): picked mon.noname-c con 0x7fedf813a480 addr v2:172.21.15.71:3300/0 2024-04-19T01:42:10.729 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fee00462640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedf80716c0 con 0x7fedf81355e0 2024-04-19T01:42:10.729 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fee00462640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:10.729 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fee00462640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fedf81360d0 con 0x7fedf80726e0 2024-04-19T01:42:10.729 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fee00462640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:10.729 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fee00462640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fedf80700b0 con 0x7fedf813a480 2024-04-19T01:42:10.729 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fee00462640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:10.729 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fee00462640 10 monclient(hunting): authenticate will time out at 916.194425s 2024-04-19T01:42:10.730 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fedfe9ff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fedf813a480 0x7fedf813c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:10.730 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fedfe9ff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fedf813a480 0x7fedf813c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52108/0 (socket says 172.21.15.71:52108) 2024-04-19T01:42:10.730 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fedfe9ff640 1 -- 172.21.15.71:0/2114734055 learned_addr learned my addr 172.21.15.71:0/2114734055 (peer_addr_for_me v2:172.21.15.71:0/0) 2024-04-19T01:42:10.730 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fedfe9ff640 10 monclient(hunting): get_auth_request con 0x7fedf813a480 auth_method 0 2024-04-19T01:42:10.730 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fedfe9ff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:10.730 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fedfe9ff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:10.730 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fedfe9ff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:10.730 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fedfd9fd640 1 --2- 172.21.15.71:0/2114734055 >> v2:172.21.15.62:3300/0 conn(0x7fedf80726e0 0x7fedf8072b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:10.730 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fedfe9ff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:10.730 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fedfe9ff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:10.730 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fedfe9ff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:10.730 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fedfd9fd640 10 monclient(hunting): get_auth_request con 0x7fedf80726e0 auth_method 0 2024-04-19T01:42:10.730 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fedfd9fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:10.731 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fedfd9fd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:10.731 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fedfd9fd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:10.731 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fedfe9ff640 10 monclient(hunting): handle_auth_done global_id 4114 payload 274 2024-04-19T01:42:10.731 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fedfe9ff640 1 -- 172.21.15.71:0/2114734055 >> v2:172.21.15.62:3300/0 conn(0x7fedf80726e0 msgr2=0x7fedf8072b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:10.731 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fedfe9ff640 1 --2- 172.21.15.71:0/2114734055 >> v2:172.21.15.62:3300/0 conn(0x7fedf80726e0 0x7fedf8072b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:10.731 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fedfe9ff640 1 -- 172.21.15.71:0/2114734055 >> v1:172.21.15.62:6789/0 conn(0x7fedf81355e0 legacy=0x7fedf81359c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:10.731 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fedfe9ff640 10 monclient: _finish_hunting 0 2024-04-19T01:42:10.731 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fedfe9ff640 1 monclient: found mon.noname-c 2024-04-19T01:42:10.731 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fedfe9ff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:10.731 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fedfe9ff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-04-19T01:42:10.731 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fedfe9ff640 1 -- 172.21.15.71:0/2114734055 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedf80719a0 con 0x7fedf813a480 2024-04-19T01:42:10.731 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.729+0000 7fedfd9fd640 1 --2- 172.21.15.71:0/2114734055 >> v2:172.21.15.62:3300/0 conn(0x7fedf80726e0 0x7fedf8072b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T01:42:10.733 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfe9ff640 1 -- 172.21.15.71:0/2114734055 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fedf813a480 msgr2=0x7fedf813c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:10.733 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfe9ff640 1 --2- 172.21.15.71:0/2114734055 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fedf813a480 0x7fedf813c870 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fedf806dc40 tx=0x7fedf001c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3ffbc2e253b593e server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:10.733 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfd1fc640 1 -- 172.21.15.71:0/2114734055 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf00a5530 con 0x7fedf813a480 2024-04-19T01:42:10.733 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfd1fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:10.733 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfd1fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:10.733 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfd1fc640 10 monclient: dump: 2024-04-19T01:42:10.734 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-04-19T01:42:10.734 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:10.734 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:10.734 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:10.734 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:10.734 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:10.734 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:10.734 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:10.734 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:10.734 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:10.734 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfd1fc640 10 monclient: _finish_auth 0 2024-04-19T01:42:10.734 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfd1fc640 10 monclient: _check_auth_tickets 2024-04-19T01:42:10.734 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfd1fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:10.734 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfd1fc640 1 -- 172.21.15.71:0/2114734055 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedf00a56d0 con 0x7fedf813a480 2024-04-19T01:42:10.734 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfd1fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:10.735 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfd1fc640 1 -- 172.21.15.71:0/2114734055 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf00a5aa0 con 0x7fedf813a480 2024-04-19T01:42:10.735 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 5 monclient: authenticate success, global_id 4114 2024-04-19T01:42:10.735 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:10.735 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 4 set_mon_vals no callback set 2024-04-19T01:42:10.735 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfd1fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:10.735 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfd1fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:42:10.735 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfd1fc640 10 monclient: dump: 2024-04-19T01:42:10.735 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-04-19T01:42:10.735 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:10.735 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:10.735 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:10.735 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:10.735 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:10.735 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:10.736 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:10.736 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:10.736 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:10.736 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 10 monclient: shutdown 2024-04-19T01:42:10.736 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:10.736 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 1 -- 172.21.15.71:0/2114734055 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fedf813a480 msgr2=0x7fedf813c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:10.736 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 1 --2- 172.21.15.71:0/2114734055 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fedf813a480 0x7fedf813c870 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fedf806dc40 tx=0x7fedf001c8b0 comp rx=0 tx=0).stop 2024-04-19T01:42:10.736 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 1 -- 172.21.15.71:0/2114734055 shutdown_connections 2024-04-19T01:42:10.736 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 1 --2- 172.21.15.71:0/2114734055 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fedf813a480 0x7fedf813c870 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:10.736 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 1 --2- 172.21.15.71:0/2114734055 >> v2:172.21.15.62:3300/0 conn(0x7fedf80726e0 0x7fedf8072b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:10.736 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 1 -- 172.21.15.71:0/2114734055 >> 172.21.15.71:0/2114734055 conn(0x7fedf80a8030 msgr2=0x7fedf80a8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:10.736 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 1 -- 172.21.15.71:0/2114734055 shutdown_connections 2024-04-19T01:42:10.736 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 1 -- 172.21.15.71:0/2114734055 wait complete. 2024-04-19T01:42:10.740 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 10 monclient: build_initial_monmap 2024-04-19T01:42:10.741 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 10 monclient: monmap: 2024-04-19T01:42:10.741 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-04-19T01:42:10.741 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:10.741 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 0.000000 2024-04-19T01:42:10.741 INFO:teuthology.orchestra.run.smithi071.stderr:created 0.000000 2024-04-19T01:42:10.741 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:10.741 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:10.741 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:10.741 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:10.741 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:10.742 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:10.742 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 1 Processor -- start 2024-04-19T01:42:10.742 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 1 -- start start 2024-04-19T01:42:10.742 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 10 monclient: init 2024-04-19T01:42:10.742 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:10.742 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-19T01:42:10.742 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fedf80726e0 0x7fedf8134090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:10.742 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 10 monclient(hunting): picked mon.noname-c con 0x7fedf80726e0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:10.742 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedf81355e0 0x7fedf809e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:10.742 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 10 monclient(hunting): picked mon.noname-a con 0x7fedf81355e0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:10.742 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fedf813a480 0x7fedf809ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:10.742 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 10 monclient(hunting): picked mon.noname-b con 0x7fedf813a480 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:10.742 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:10.742 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fedf813f960 con 0x7fedf81355e0 2024-04-19T01:42:10.743 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:10.743 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fedf813f7e0 con 0x7fedf813a480 2024-04-19T01:42:10.743 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:10.743 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fedf813fae0 con 0x7fedf80726e0 2024-04-19T01:42:10.743 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:10.743 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fee00462640 10 monclient(hunting): authenticate will time out at 916.203338s 2024-04-19T01:42:10.743 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfe9ff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fedf813a480 0x7fedf809ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:10.743 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfe9ff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fedf813a480 0x7fedf809ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52116/0 (socket says 172.21.15.71:52116) 2024-04-19T01:42:10.743 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfe9ff640 1 -- 172.21.15.71:0/1494319771 learned_addr learned my addr 172.21.15.71:0/1494319771 (peer_addr_for_me v2:172.21.15.71:0/0) 2024-04-19T01:42:10.743 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfe9ff640 10 monclient(hunting): get_auth_request con 0x7fedf813a480 auth_method 0 2024-04-19T01:42:10.743 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfe9ff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:10.743 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfe9ff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:10.743 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfe9ff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:10.743 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfe9ff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:10.743 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfe9ff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:10.743 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfe9ff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:10.744 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfd9fd640 1 --2- 172.21.15.71:0/1494319771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedf81355e0 0x7fedf809e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:10.744 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfe1fe640 1 --2- 172.21.15.71:0/1494319771 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fedf80726e0 0x7fedf8134090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:10.744 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfd9fd640 10 monclient(hunting): get_auth_request con 0x7fedf81355e0 auth_method 0 2024-04-19T01:42:10.744 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfd9fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:10.744 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfd9fd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:10.744 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfd9fd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:10.744 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfe9ff640 10 monclient(hunting): handle_auth_done global_id 4117 payload 824 2024-04-19T01:42:10.744 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfe9ff640 1 -- 172.21.15.71:0/1494319771 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fedf80726e0 msgr2=0x7fedf8134090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:10.744 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfe9ff640 1 --2- 172.21.15.71:0/1494319771 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fedf80726e0 0x7fedf8134090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:10.744 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfe9ff640 1 -- 172.21.15.71:0/1494319771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedf81355e0 msgr2=0x7fedf809e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:10.744 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfe9ff640 1 --2- 172.21.15.71:0/1494319771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedf81355e0 0x7fedf809e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:10.744 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfe9ff640 10 monclient: _finish_hunting 0 2024-04-19T01:42:10.744 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfe9ff640 1 monclient: found mon.noname-b 2024-04-19T01:42:10.744 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfe9ff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:10.744 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfe9ff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.71:3300/0 2024-04-19T01:42:10.744 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.737+0000 7fedfe9ff640 1 -- 172.21.15.71:0/1494319771 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedf811ea40 con 0x7fedf813a480 2024-04-19T01:42:10.745 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.741+0000 7fedfe9ff640 1 -- 172.21.15.71:0/1494319771 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fedf813a480 msgr2=0x7fedf809ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:10.745 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.741+0000 7fedfe9ff640 1 --2- 172.21.15.71:0/1494319771 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fedf813a480 0x7fedf809ea40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fedf001d150 tx=0x7fedf0019650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:10.745 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.741+0000 7fedeeffd640 1 -- 172.21.15.71:0/1494319771 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf00bc020 con 0x7fedf813a480 2024-04-19T01:42:10.745 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.741+0000 7fedeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:10.745 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.741+0000 7fedeeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:42:10.745 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.741+0000 7fedeeffd640 10 monclient: dump: 2024-04-19T01:42:10.745 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-04-19T01:42:10.745 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:10.745 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:10.745 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:10.745 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:10.745 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:10.745 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:10.745 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:10.745 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:10.745 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:10.746 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.741+0000 7fedeeffd640 10 monclient: _finish_auth 0 2024-04-19T01:42:10.746 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.741+0000 7fedeeffd640 10 monclient: _check_auth_tickets 2024-04-19T01:42:10.746 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.741+0000 7fedeeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:10.746 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.741+0000 7fedeeffd640 1 -- 172.21.15.71:0/1494319771 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedf00b8a90 con 0x7fedf813a480 2024-04-19T01:42:10.746 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.741+0000 7fedeeffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:10.746 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.741+0000 7fedeeffd640 1 -- 172.21.15.71:0/1494319771 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf00a5530 con 0x7fedf813a480 2024-04-19T01:42:10.746 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.741+0000 7fedeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:10.746 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.741+0000 7fedeeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:42:10.746 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.741+0000 7fedeeffd640 10 monclient: dump: 2024-04-19T01:42:10.746 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-04-19T01:42:10.746 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:10.746 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:10.746 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:10.746 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:10.746 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:10.746 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:10.747 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:10.747 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:10.747 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:10.747 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.741+0000 7fedfc9fb640 4 set_mon_vals no callback set 2024-04-19T01:42:10.747 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.741+0000 7fee00462640 5 monclient: authenticate success, global_id 4117 2024-04-19T01:42:10.747 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.741+0000 7fee00462640 10 monclient: _renew_subs 2024-04-19T01:42:10.747 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.741+0000 7fee00462640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:42:10.747 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.741+0000 7fee00462640 1 -- 172.21.15.71:0/1494319771 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedf80a0d00 con 0x7fedf813a480 2024-04-19T01:42:10.747 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.741+0000 7fee00462640 10 monclient: _renew_subs 2024-04-19T01:42:10.747 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.741+0000 7fee00462640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:42:10.747 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.741+0000 7fee00462640 1 -- 172.21.15.71:0/1494319771 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedf80731b0 con 0x7fedf813a480 2024-04-19T01:42:10.748 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.741+0000 7fedeeffd640 1 -- 172.21.15.71:0/1494319771 <== mon.1 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fedf00bf020 con 0x7fedf813a480 2024-04-19T01:42:10.748 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.741+0000 7fedeeffd640 1 -- 172.21.15.71:0/1494319771 <== mon.1 v2:172.21.15.71:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7fedf00b67f0 con 0x7fedf813a480 2024-04-19T01:42:10.748 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.741+0000 7fee00462640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:10.748 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.741+0000 7fee00462640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:10.748 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.741+0000 7fee00462640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:42:10.748 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.741+0000 7fee00462640 1 -- 172.21.15.71:0/1494319771 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedf8068ab0 con 0x7fedf813a480 2024-04-19T01:42:10.748 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.745+0000 7fedeeffd640 1 -- 172.21.15.71:0/1494319771 <== mon.1 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fedf00c6030 con 0x7fedf813a480 2024-04-19T01:42:10.748 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.745+0000 7fedeeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:10.748 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.745+0000 7fedeeffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:10.810 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.813+0000 7fee00462640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "dec72cc0-fa6d-4fe9-b9cd-19dec591ab68", "id": 1}] 2024-04-19T01:42:10.810 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.813+0000 7fee00462640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "dec72cc0-fa6d-4fe9-b9cd-19dec591ab68", "id": 1}] 2024-04-19T01:42:10.810 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.813+0000 7fee00462640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:42:10.810 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.813+0000 7fee00462640 1 -- 172.21.15.71:0/1494319771 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd new", "uuid": "dec72cc0-fa6d-4fe9-b9cd-19dec591ab68", "id": 1} v 0) -- 0x7fedf811e820 con 0x7fedf813a480 2024-04-19T01:42:10.816 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.821+0000 7fedeeffd640 1 -- 172.21.15.71:0/1494319771 <== mon.1 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dec72cc0-fa6d-4fe9-b9cd-19dec591ab68", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7fedf00165c0 con 0x7fedf813a480 2024-04-19T01:42:10.816 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.821+0000 7fedeeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "dec72cc0-fa6d-4fe9-b9cd-19dec591ab68", "id": 1}] 2024-04-19T01:42:10.816 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.821+0000 7fedeeffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:10.816 INFO:teuthology.orchestra.run.smithi071.stdout:1 2024-04-19T01:42:10.818 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.821+0000 7fee00462640 10 monclient: shutdown 2024-04-19T01:42:10.818 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.821+0000 7fee00462640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:10.818 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.821+0000 7fee00462640 1 -- 172.21.15.71:0/1494319771 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fedf813a480 msgr2=0x7fedf809ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:10.818 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.821+0000 7fee00462640 1 --2- 172.21.15.71:0/1494319771 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fedf813a480 0x7fedf809ea40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fedf001d150 tx=0x7fedf0019650 comp rx=0 tx=0).stop 2024-04-19T01:42:10.818 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.821+0000 7fee00462640 1 -- 172.21.15.71:0/1494319771 shutdown_connections 2024-04-19T01:42:10.818 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.821+0000 7fee00462640 1 --2- 172.21.15.71:0/1494319771 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fedf813a480 0x7fedf809ea40 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:10.818 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.821+0000 7fee00462640 1 --2- 172.21.15.71:0/1494319771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedf81355e0 0x7fedf809e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:10.818 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.821+0000 7fee00462640 1 --2- 172.21.15.71:0/1494319771 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fedf80726e0 0x7fedf8134090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:10.818 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.821+0000 7fee00462640 1 -- 172.21.15.71:0/1494319771 >> 172.21.15.71:0/1494319771 conn(0x7fedf80a8030 msgr2=0x7fedf80a5960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:10.818 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.821+0000 7fee00462640 1 -- 172.21.15.71:0/1494319771 shutdown_connections 2024-04-19T01:42:10.818 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.821+0000 7fee00462640 1 -- 172.21.15.71:0/1494319771 wait complete. 2024-04-19T01:42:10.834 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd new 2045906a-e0fd-4c41-b73c-27bf7103243b 2 2024-04-19T01:42:10.859 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-04-19T01:42:10.865+0000 7f76bc1b9180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-19T01:42:10.954 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.957+0000 7fb427b0c640 10 monclient: get_monmap_and_config 2024-04-19T01:42:10.954 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.957+0000 7fb427b0c640 10 monclient: build_initial_monmap 2024-04-19T01:42:10.954 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.957+0000 7fb427b0c640 10 monclient: monmap: 2024-04-19T01:42:10.954 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-04-19T01:42:10.954 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:10.954 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-04-19T01:42:10.959865+0000 2024-04-19T01:42:10.954 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-04-19T01:42:10.959865+0000 2024-04-19T01:42:10.954 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:10.954 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:10.954 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:10.955 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:10.955 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:10.955 INFO:teuthology.orchestra.run.smithi071.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:10.955 INFO:teuthology.orchestra.run.smithi071.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:10.955 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:10.957 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.957+0000 7fb427b0c640 1 Processor -- start 2024-04-19T01:42:10.957 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.957+0000 7fb427b0c640 1 -- start start 2024-04-19T01:42:10.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.957+0000 7fb427b0c640 10 monclient: init 2024-04-19T01:42:10.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.957+0000 7fb427b0c640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:10.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.957+0000 7fb427b0c640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-04-19T01:42:10.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.957+0000 7fb427b0c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb420137780 0x7fb420137b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:10.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.957+0000 7fb427b0c640 10 monclient(hunting): picked mon.noname-a con 0x7fb420137780 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:10.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.957+0000 7fb427b0c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb4201380a0 0x7fb4200a45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:10.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.957+0000 7fb427b0c640 10 monclient(hunting): picked mon.noname-c con 0x7fb4201380a0 addr v2:172.21.15.71:3300/0 2024-04-19T01:42:10.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.957+0000 7fb427b0c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb4200a60b0 0x7fb4200a4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:10.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.957+0000 7fb427b0c640 10 monclient(hunting): picked mon.noname-b con 0x7fb4200a60b0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:10.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.957+0000 7fb427b0c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:10.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.957+0000 7fb427b0c640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fb42009bd20 con 0x7fb420137780 2024-04-19T01:42:10.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.957+0000 7fb427b0c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:10.958 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.957+0000 7fb427b0c640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fb42009c2e0 con 0x7fb4201380a0 2024-04-19T01:42:10.959 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.957+0000 7fb427b0c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:10.959 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.957+0000 7fb427b0c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fb42009c000 con 0x7fb4200a60b0 2024-04-19T01:42:10.959 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.957+0000 7fb427b0c640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:10.959 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.957+0000 7fb427b0c640 10 monclient(hunting): authenticate will time out at 916.423674s 2024-04-19T01:42:10.959 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.957+0000 7fb4250a7640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb4201380a0 0x7fb4200a45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:10.959 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.957+0000 7fb4250a7640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb4201380a0 0x7fb4200a45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52118/0 (socket says 172.21.15.71:52118) 2024-04-19T01:42:10.959 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.957+0000 7fb4250a7640 1 -- 172.21.15.71:0/3987223272 learned_addr learned my addr 172.21.15.71:0/3987223272 (peer_addr_for_me v2:172.21.15.71:0/0) 2024-04-19T01:42:10.959 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.957+0000 7fb4250a7640 10 monclient(hunting): get_auth_request con 0x7fb4201380a0 auth_method 0 2024-04-19T01:42:10.959 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.957+0000 7fb4250a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:10.959 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.957+0000 7fb4250a7640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:10.959 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.957+0000 7fb4250a7640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:10.959 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.961+0000 7fb4250a7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:10.959 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.961+0000 7fb4250a7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:10.959 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.961+0000 7fb4250a7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:10.960 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.961+0000 7fb4260a9640 1 --2- 172.21.15.71:0/3987223272 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb4200a60b0 0x7fb4200a4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:10.960 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.961+0000 7fb4258a8640 1 --2- 172.21.15.71:0/3987223272 >> v2:172.21.15.62:3300/0 conn(0x7fb420137780 0x7fb420137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:10.960 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.961+0000 7fb4260a9640 10 monclient(hunting): get_auth_request con 0x7fb4200a60b0 auth_method 0 2024-04-19T01:42:10.960 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.961+0000 7fb4260a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:10.960 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.961+0000 7fb4260a9640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:10.960 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.961+0000 7fb4260a9640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:10.960 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.961+0000 7fb4258a8640 10 monclient(hunting): get_auth_request con 0x7fb420137780 auth_method 0 2024-04-19T01:42:10.960 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.961+0000 7fb4258a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:10.960 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.961+0000 7fb4258a8640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:10.960 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.961+0000 7fb4258a8640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:10.960 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.961+0000 7fb4250a7640 10 monclient(hunting): handle_auth_done global_id 4120 payload 274 2024-04-19T01:42:10.960 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.961+0000 7fb4250a7640 1 -- 172.21.15.71:0/3987223272 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb4200a60b0 msgr2=0x7fb4200a4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:10.960 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.961+0000 7fb4250a7640 1 --2- 172.21.15.71:0/3987223272 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb4200a60b0 0x7fb4200a4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:10.960 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.961+0000 7fb4250a7640 1 -- 172.21.15.71:0/3987223272 >> v2:172.21.15.62:3300/0 conn(0x7fb420137780 msgr2=0x7fb420137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:10.960 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.961+0000 7fb4250a7640 1 --2- 172.21.15.71:0/3987223272 >> v2:172.21.15.62:3300/0 conn(0x7fb420137780 0x7fb420137b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:10.961 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.961+0000 7fb4250a7640 10 monclient: _finish_hunting 0 2024-04-19T01:42:10.961 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.961+0000 7fb4250a7640 1 monclient: found mon.noname-c 2024-04-19T01:42:10.961 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.961+0000 7fb4250a7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:10.961 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.961+0000 7fb4250a7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-04-19T01:42:10.961 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.961+0000 7fb4250a7640 1 -- 172.21.15.71:0/3987223272 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb420073210 con 0x7fb4201380a0 2024-04-19T01:42:10.965 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4250a7640 1 -- 172.21.15.71:0/3987223272 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4201380a0 msgr2=0x7fb4200a45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:10.965 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4250a7640 1 --2- 172.21.15.71:0/3987223272 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4201380a0 0x7fb4200a45d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb4200a74f0 tx=0x7fb41401c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=acd8fcec8028e05a server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:10.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4248a6640 1 -- 172.21.15.71:0/3987223272 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4140b0070 con 0x7fb4201380a0 2024-04-19T01:42:10.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4248a6640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:10.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4248a6640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:10.966 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4248a6640 10 monclient: dump: 2024-04-19T01:42:10.966 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-04-19T01:42:10.966 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:10.966 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:10.966 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:10.966 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:10.966 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:10.966 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:10.966 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:10.967 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:10.967 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:10.967 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4248a6640 10 monclient: _finish_auth 0 2024-04-19T01:42:10.967 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4248a6640 10 monclient: _check_auth_tickets 2024-04-19T01:42:10.967 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4248a6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:10.967 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4248a6640 1 -- 172.21.15.71:0/3987223272 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4140ab600 con 0x7fb4201380a0 2024-04-19T01:42:10.967 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4248a6640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:10.967 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4248a6640 1 -- 172.21.15.71:0/3987223272 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4140ab9d0 con 0x7fb4201380a0 2024-04-19T01:42:10.967 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 5 monclient: authenticate success, global_id 4120 2024-04-19T01:42:10.967 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:10.967 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 4 set_mon_vals no callback set 2024-04-19T01:42:10.967 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4248a6640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:10.967 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4248a6640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:42:10.967 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4248a6640 10 monclient: dump: 2024-04-19T01:42:10.968 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-04-19T01:42:10.968 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:10.968 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:10.968 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:10.968 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:10.968 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:10.968 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:10.968 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:10.968 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:10.968 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:10.968 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 10 monclient: shutdown 2024-04-19T01:42:10.968 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:10.968 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 1 -- 172.21.15.71:0/3987223272 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4201380a0 msgr2=0x7fb4200a45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:10.968 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 1 --2- 172.21.15.71:0/3987223272 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4201380a0 0x7fb4200a45d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb4200a74f0 tx=0x7fb41401c3e0 comp rx=0 tx=0).stop 2024-04-19T01:42:10.968 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 1 -- 172.21.15.71:0/3987223272 shutdown_connections 2024-04-19T01:42:10.969 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 1 --2- 172.21.15.71:0/3987223272 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb4200a60b0 0x7fb4200a4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:10.969 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 1 --2- 172.21.15.71:0/3987223272 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4201380a0 0x7fb4200a45d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:10.969 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 1 --2- 172.21.15.71:0/3987223272 >> v2:172.21.15.62:3300/0 conn(0x7fb420137780 0x7fb420137b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:10.969 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 1 -- 172.21.15.71:0/3987223272 >> 172.21.15.71:0/3987223272 conn(0x7fb4200aa1d0 msgr2=0x7fb4200a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:10.969 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 1 -- 172.21.15.71:0/3987223272 shutdown_connections 2024-04-19T01:42:10.969 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 1 -- 172.21.15.71:0/3987223272 wait complete. 2024-04-19T01:42:10.969 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 10 monclient: build_initial_monmap 2024-04-19T01:42:10.969 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 10 monclient: monmap: 2024-04-19T01:42:10.969 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-04-19T01:42:10.969 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:10.969 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 0.000000 2024-04-19T01:42:10.969 INFO:teuthology.orchestra.run.smithi071.stderr:created 0.000000 2024-04-19T01:42:10.969 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:10.970 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:10.970 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:10.970 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:10.970 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:10.970 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:10.970 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 1 Processor -- start 2024-04-19T01:42:10.970 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 1 -- start start 2024-04-19T01:42:10.970 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 10 monclient: init 2024-04-19T01:42:10.970 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:10.970 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-19T01:42:10.970 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4200a60b0 0x7fb420120560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:10.970 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 10 monclient(hunting): picked mon.noname-b con 0x7fb4200a60b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:10.970 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb420137780 0x7fb4200a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:10.970 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 10 monclient(hunting): picked mon.noname-c con 0x7fb420137780 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:10.971 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4201380a0 0x7fb420136350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:10.971 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 10 monclient(hunting): picked mon.noname-a con 0x7fb4201380a0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:10.971 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:10.971 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb420141160 con 0x7fb4201380a0 2024-04-19T01:42:10.971 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:10.971 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb4200a5460 con 0x7fb4200a60b0 2024-04-19T01:42:10.971 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:10.971 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fb4201412e0 con 0x7fb420137780 2024-04-19T01:42:10.971 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:10.971 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb427b0c640 10 monclient(hunting): authenticate will time out at 916.430890s 2024-04-19T01:42:10.971 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4258a8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4200a60b0 0x7fb420120560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:10.971 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4258a8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4200a60b0 0x7fb420120560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52120/0 (socket says 172.21.15.71:52120) 2024-04-19T01:42:10.971 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4258a8640 1 -- 172.21.15.71:0/3943378364 learned_addr learned my addr 172.21.15.71:0/3943378364 (peer_addr_for_me v2:172.21.15.71:0/0) 2024-04-19T01:42:10.971 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4258a8640 10 monclient(hunting): get_auth_request con 0x7fb4200a60b0 auth_method 0 2024-04-19T01:42:10.972 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4258a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:10.972 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4258a8640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:10.972 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4258a8640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:10.972 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4258a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:10.972 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4258a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:10.972 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4258a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:10.972 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4250a7640 1 --2- 172.21.15.71:0/3943378364 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb420137780 0x7fb4200a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:10.972 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4260a9640 1 --2- 172.21.15.71:0/3943378364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4201380a0 0x7fb420136350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:10.972 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4250a7640 10 monclient(hunting): get_auth_request con 0x7fb420137780 auth_method 0 2024-04-19T01:42:10.972 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4250a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:10.972 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4250a7640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:10.972 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4250a7640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:10.972 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4258a8640 10 monclient(hunting): handle_auth_done global_id 4123 payload 824 2024-04-19T01:42:10.972 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4258a8640 1 -- 172.21.15.71:0/3943378364 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb420137780 msgr2=0x7fb4200a0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:10.972 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4258a8640 1 --2- 172.21.15.71:0/3943378364 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb420137780 0x7fb4200a0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:10.973 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4258a8640 1 -- 172.21.15.71:0/3943378364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4201380a0 msgr2=0x7fb420136350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:10.973 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4258a8640 1 --2- 172.21.15.71:0/3943378364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4201380a0 0x7fb420136350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:10.973 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4258a8640 10 monclient: _finish_hunting 0 2024-04-19T01:42:10.973 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4258a8640 1 monclient: found mon.noname-b 2024-04-19T01:42:10.973 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4258a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:10.973 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4258a8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.71:3300/0 2024-04-19T01:42:10.973 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4258a8640 1 -- 172.21.15.71:0/3943378364 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb42013de80 con 0x7fb4200a60b0 2024-04-19T01:42:10.973 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4260a9640 10 monclient: get_auth_request con 0x7fb4201380a0 auth_method 0 2024-04-19T01:42:10.973 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.965+0000 7fb4260a9640 1 --2- 172.21.15.71:0/3943378364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4201380a0 0x7fb420136350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:42:10.973 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.969+0000 7fb4258a8640 1 -- 172.21.15.71:0/3943378364 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4200a60b0 msgr2=0x7fb420120560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:10.973 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.969+0000 7fb4258a8640 1 --2- 172.21.15.71:0/3943378364 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4200a60b0 0x7fb420120560 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb41800d800 tx=0x7fb41800dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:10.973 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.969+0000 7fb41e7fc640 1 -- 172.21.15.71:0/3943378364 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb41800b8d0 con 0x7fb4200a60b0 2024-04-19T01:42:10.973 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.969+0000 7fb41e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:10.974 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.969+0000 7fb41e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:42:10.974 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.969+0000 7fb41e7fc640 10 monclient: dump: 2024-04-19T01:42:10.974 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-04-19T01:42:10.974 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:10.974 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:10.974 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:10.974 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:10.974 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:10.974 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:10.974 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:10.974 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:10.974 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:10.974 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.969+0000 7fb41e7fc640 10 monclient: _finish_auth 0 2024-04-19T01:42:10.974 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.969+0000 7fb41e7fc640 10 monclient: _check_auth_tickets 2024-04-19T01:42:10.975 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.969+0000 7fb41e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:10.975 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.969+0000 7fb41e7fc640 1 -- 172.21.15.71:0/3943378364 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb418019070 con 0x7fb4200a60b0 2024-04-19T01:42:10.975 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.969+0000 7fb427b0c640 5 monclient: authenticate success, global_id 4123 2024-04-19T01:42:10.975 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.969+0000 7fb427b0c640 10 monclient: _renew_subs 2024-04-19T01:42:10.975 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.969+0000 7fb427b0c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:42:10.975 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.969+0000 7fb427b0c640 1 -- 172.21.15.71:0/3943378364 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4200a24d0 con 0x7fb4200a60b0 2024-04-19T01:42:10.975 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.969+0000 7fb41e7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:10.975 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.969+0000 7fb41e7fc640 1 -- 172.21.15.71:0/3943378364 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb41800bc70 con 0x7fb4200a60b0 2024-04-19T01:42:10.975 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.969+0000 7fb41e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:10.975 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.969+0000 7fb41e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:42:10.975 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.969+0000 7fb41e7fc640 10 monclient: dump: 2024-04-19T01:42:10.975 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-04-19T01:42:10.975 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:10.975 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:10.975 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:10.975 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:10.976 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:10.976 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:10.976 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:10.976 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:10.976 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:10.976 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.969+0000 7fb41ffff640 4 set_mon_vals no callback set 2024-04-19T01:42:10.976 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.969+0000 7fb427b0c640 10 monclient: _renew_subs 2024-04-19T01:42:10.976 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.969+0000 7fb427b0c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:42:10.976 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.969+0000 7fb427b0c640 1 -- 172.21.15.71:0/3943378364 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb420121450 con 0x7fb4200a60b0 2024-04-19T01:42:10.976 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.969+0000 7fb41e7fc640 1 -- 172.21.15.71:0/3943378364 <== mon.1 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb41801e460 con 0x7fb4200a60b0 2024-04-19T01:42:10.976 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.969+0000 7fb41e7fc640 1 -- 172.21.15.71:0/3943378364 <== mon.1 v2:172.21.15.71:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7fb418014ad0 con 0x7fb4200a60b0 2024-04-19T01:42:10.977 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.969+0000 7fb427b0c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:10.977 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.969+0000 7fb427b0c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:10.977 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.969+0000 7fb427b0c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:42:10.977 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.969+0000 7fb427b0c640 1 -- 172.21.15.71:0/3943378364 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4200a1d40 con 0x7fb4200a60b0 2024-04-19T01:42:10.977 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.973+0000 7fb41e7fc640 1 -- 172.21.15.71:0/3943378364 <== mon.1 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fb418014cd0 con 0x7fb4200a60b0 2024-04-19T01:42:10.977 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.973+0000 7fb41e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:10.977 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:10.973+0000 7fb41e7fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:11.086 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.089+0000 7fb427b0c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2045906a-e0fd-4c41-b73c-27bf7103243b", "id": 2}] 2024-04-19T01:42:11.086 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.089+0000 7fb427b0c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2045906a-e0fd-4c41-b73c-27bf7103243b", "id": 2}] 2024-04-19T01:42:11.086 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.089+0000 7fb427b0c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:42:11.086 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.089+0000 7fb427b0c640 1 -- 172.21.15.71:0/3943378364 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2045906a-e0fd-4c41-b73c-27bf7103243b", "id": 2} v 0) -- 0x7fb420068ab0 con 0x7fb4200a60b0 2024-04-19T01:42:11.092 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-04-19T01:42:11.097+0000 7f76bc1b9180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-19T01:42:11.093 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.097+0000 7fb41e7fc640 1 -- 172.21.15.71:0/3943378364 <== mon.1 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2045906a-e0fd-4c41-b73c-27bf7103243b", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7fb4180146b0 con 0x7fb4200a60b0 2024-04-19T01:42:11.093 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.097+0000 7fb41e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2045906a-e0fd-4c41-b73c-27bf7103243b", "id": 2}] 2024-04-19T01:42:11.093 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.097+0000 7fb41e7fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:11.093 INFO:teuthology.orchestra.run.smithi071.stdout:2 2024-04-19T01:42:11.095 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.097+0000 7fb427b0c640 10 monclient: shutdown 2024-04-19T01:42:11.095 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.097+0000 7fb427b0c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:11.095 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.097+0000 7fb427b0c640 1 -- 172.21.15.71:0/3943378364 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4200a60b0 msgr2=0x7fb420120560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:11.095 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.097+0000 7fb427b0c640 1 --2- 172.21.15.71:0/3943378364 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4200a60b0 0x7fb420120560 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb41800d800 tx=0x7fb41800dcc0 comp rx=0 tx=0).stop 2024-04-19T01:42:11.095 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.097+0000 7fb427b0c640 1 -- 172.21.15.71:0/3943378364 shutdown_connections 2024-04-19T01:42:11.095 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.097+0000 7fb427b0c640 1 --2- 172.21.15.71:0/3943378364 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb420137780 0x7fb4200a0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:11.095 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.097+0000 7fb427b0c640 1 --2- 172.21.15.71:0/3943378364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb4201380a0 0x7fb420136350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:11.095 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.097+0000 7fb427b0c640 1 --2- 172.21.15.71:0/3943378364 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4200a60b0 0x7fb420120560 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:11.096 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.097+0000 7fb427b0c640 1 -- 172.21.15.71:0/3943378364 >> 172.21.15.71:0/3943378364 conn(0x7fb4200aa1d0 msgr2=0x7fb4200a83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:11.096 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.097+0000 7fb427b0c640 1 -- 172.21.15.71:0/3943378364 shutdown_connections 2024-04-19T01:42:11.096 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.097+0000 7fb427b0c640 1 -- 172.21.15.71:0/3943378364 wait complete. 2024-04-19T01:42:11.115 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd new 230f6e15-548c-4338-a8ec-067b8f85bd2e 3 2024-04-19T01:42:11.272 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17de10c640 10 monclient: get_monmap_and_config 2024-04-19T01:42:11.272 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17de10c640 10 monclient: build_initial_monmap 2024-04-19T01:42:11.273 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17de10c640 10 monclient: monmap: 2024-04-19T01:42:11.273 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-04-19T01:42:11.273 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:11.273 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-04-19T01:42:11.278546+0000 2024-04-19T01:42:11.273 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-04-19T01:42:11.278546+0000 2024-04-19T01:42:11.273 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:11.273 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:11.273 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:11.273 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:11.273 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:11.273 INFO:teuthology.orchestra.run.smithi071.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:11.273 INFO:teuthology.orchestra.run.smithi071.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:11.274 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:11.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17de10c640 1 Processor -- start 2024-04-19T01:42:11.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17de10c640 1 -- start start 2024-04-19T01:42:11.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17de10c640 10 monclient: init 2024-04-19T01:42:11.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17de10c640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:11.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17de10c640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-04-19T01:42:11.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17de10c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f17d8137830 0x7f17d8137c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:11.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17de10c640 10 monclient(hunting): picked mon.noname-b con 0x7f17d8137830 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:11.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17de10c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f17d80a0b60 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:11.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17de10c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f17d80a1650 addr v1:172.21.15.71:6789/0 2024-04-19T01:42:11.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17de10c640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17d809a270 con 0x7f17d80a0b60 2024-04-19T01:42:11.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17de10c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17d8138150 con 0x7f17d80a1650 2024-04-19T01:42:11.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17de10c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:11.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17de10c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f17d80a2a30 con 0x7f17d8137830 2024-04-19T01:42:11.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17de10c640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:11.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17de10c640 10 monclient(hunting): authenticate will time out at 916.742815s 2024-04-19T01:42:11.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17d7fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f17d80a1650 0x7f17d813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:47924/0 (socket says 172.21.15.71:47924) 2024-04-19T01:42:11.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17d7fff640 1 -- 172.21.15.71:0/693586428 learned_addr learned my addr 172.21.15.71:0/693586428 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-04-19T01:42:11.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17d77fe640 1 --2- 172.21.15.71:0/693586428 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f17d8137830 0x7f17d8137c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:11.278 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17d77fe640 10 monclient(hunting): get_auth_request con 0x7f17d8137830 auth_method 0 2024-04-19T01:42:11.278 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:11.278 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17d77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:11.278 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:11.278 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:11.278 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:11.278 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:11.278 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17d77fe640 10 monclient(hunting): handle_auth_done global_id 4121 payload 274 2024-04-19T01:42:11.278 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17d77fe640 1 -- 172.21.15.71:0/693586428 >> v1:172.21.15.71:6789/0 conn(0x7f17d80a1650 legacy=0x7f17d813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:11.278 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17d77fe640 1 -- 172.21.15.71:0/693586428 >> v1:172.21.15.62:6789/0 conn(0x7f17d80a0b60 legacy=0x7f17d80a0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:11.278 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17d77fe640 10 monclient: _finish_hunting 0 2024-04-19T01:42:11.278 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17d77fe640 1 monclient: found mon.noname-b 2024-04-19T01:42:11.278 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:11.278 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17d77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:42:11.278 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.277+0000 7f17d77fe640 1 -- 172.21.15.71:0/693586428 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17d80a20b0 con 0x7f17d8137830 2024-04-19T01:42:11.279 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17d77fe640 1 -- 172.21.15.71:0/693586428 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f17d8137830 msgr2=0x7f17d8137c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:11.279 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17d77fe640 1 --2- 172.21.15.71:0/693586428 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f17d8137830 0x7f17d8137c10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f17cc0097f0 tx=0x7f17cc01c430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af86c05a31646940 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:11.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17d67fc640 1 -- 172.21.15.71:0/693586428 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17cc0b0070 con 0x7f17d8137830 2024-04-19T01:42:11.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:11.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17d67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:42:11.280 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17d67fc640 10 monclient: dump: 2024-04-19T01:42:11.280 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-04-19T01:42:11.280 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:11.281 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:11.281 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:11.281 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:11.281 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:11.281 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:11.281 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:11.281 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:11.281 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:11.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17d67fc640 10 monclient: _finish_auth 0 2024-04-19T01:42:11.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17d67fc640 10 monclient: _check_auth_tickets 2024-04-19T01:42:11.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:11.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17d67fc640 1 -- 172.21.15.71:0/693586428 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17cc0ab600 con 0x7f17d8137830 2024-04-19T01:42:11.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17d67fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:11.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17d67fc640 1 -- 172.21.15.71:0/693586428 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17cc0ab9d0 con 0x7f17d8137830 2024-04-19T01:42:11.281 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 5 monclient: authenticate success, global_id 4121 2024-04-19T01:42:11.282 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:11.282 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 4 set_mon_vals no callback set 2024-04-19T01:42:11.282 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:11.282 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17d67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:11.282 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17d67fc640 10 monclient: dump: 2024-04-19T01:42:11.282 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-04-19T01:42:11.282 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:11.282 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:11.282 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:11.282 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:11.282 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:11.282 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:11.282 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:11.282 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:11.283 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:11.287 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 10 monclient: shutdown 2024-04-19T01:42:11.287 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:11.287 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 1 -- 172.21.15.71:0/693586428 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f17d8137830 msgr2=0x7f17d8137c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:11.288 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 1 --2- 172.21.15.71:0/693586428 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f17d8137830 0x7f17d8137c10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f17cc0097f0 tx=0x7f17cc01c430 comp rx=0 tx=0).stop 2024-04-19T01:42:11.288 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 1 -- 172.21.15.71:0/693586428 shutdown_connections 2024-04-19T01:42:11.288 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 1 --2- 172.21.15.71:0/693586428 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f17d8137830 0x7f17d8137c10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:11.288 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 1 -- 172.21.15.71:0/693586428 >> 172.21.15.71:0/693586428 conn(0x7f17d80a6160 msgr2=0x7f17d80a6540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:11.288 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 1 -- 172.21.15.71:0/693586428 shutdown_connections 2024-04-19T01:42:11.288 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 1 -- 172.21.15.71:0/693586428 wait complete. 2024-04-19T01:42:11.288 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 10 monclient: build_initial_monmap 2024-04-19T01:42:11.288 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 10 monclient: monmap: 2024-04-19T01:42:11.288 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-04-19T01:42:11.288 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:11.288 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 0.000000 2024-04-19T01:42:11.288 INFO:teuthology.orchestra.run.smithi071.stderr:created 0.000000 2024-04-19T01:42:11.289 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:11.289 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:11.289 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:11.289 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:11.289 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:11.289 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:11.289 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 1 Processor -- start 2024-04-19T01:42:11.289 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 1 -- start start 2024-04-19T01:42:11.289 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 10 monclient: init 2024-04-19T01:42:11.289 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:11.289 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-19T01:42:11.289 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17d80a0b60 0x7f17d81cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:11.289 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 10 monclient(hunting): picked mon.noname-b con 0x7f17d80a0b60 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:11.289 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f17d80a1650 0x7f17d8135430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:11.289 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 10 monclient(hunting): picked mon.noname-c con 0x7f17d80a1650 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:11.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17d8137830 0x7f17d8135970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:11.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 10 monclient(hunting): picked mon.noname-a con 0x7f17d8137830 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:11.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:11.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8143dd0 con 0x7f17d8137830 2024-04-19T01:42:11.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:11.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f17d809a550 con 0x7f17d80a0b60 2024-04-19T01:42:11.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:11.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f17d8143f50 con 0x7f17d80a1650 2024-04-19T01:42:11.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:11.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17de10c640 10 monclient(hunting): authenticate will time out at 916.751727s 2024-04-19T01:42:11.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17d77fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17d80a0b60 0x7f17d81cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:11.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17d77fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17d80a0b60 0x7f17d81cffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52128/0 (socket says 172.21.15.71:52128) 2024-04-19T01:42:11.290 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17d77fe640 1 -- 172.21.15.71:0/1810305081 learned_addr learned my addr 172.21.15.71:0/1810305081 (peer_addr_for_me v2:172.21.15.71:0/0) 2024-04-19T01:42:11.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17d77fe640 10 monclient(hunting): get_auth_request con 0x7f17d80a0b60 auth_method 0 2024-04-19T01:42:11.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:11.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17d77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:11.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:11.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:11.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:11.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.285+0000 7f17d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:11.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17d6ffd640 1 --2- 172.21.15.71:0/1810305081 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f17d80a1650 0x7f17d8135430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:11.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17d7fff640 1 --2- 172.21.15.71:0/1810305081 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17d8137830 0x7f17d8135970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:11.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17d7fff640 10 monclient(hunting): get_auth_request con 0x7f17d8137830 auth_method 0 2024-04-19T01:42:11.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:11.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17d7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:11.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:11.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17d6ffd640 10 monclient(hunting): get_auth_request con 0x7f17d80a1650 auth_method 0 2024-04-19T01:42:11.291 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:11.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17d6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:11.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:11.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17d77fe640 10 monclient(hunting): handle_auth_done global_id 4126 payload 824 2024-04-19T01:42:11.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17d77fe640 1 -- 172.21.15.71:0/1810305081 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f17d80a1650 msgr2=0x7f17d8135430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:11.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17d77fe640 1 --2- 172.21.15.71:0/1810305081 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f17d80a1650 0x7f17d8135430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:11.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17d77fe640 1 -- 172.21.15.71:0/1810305081 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17d8137830 msgr2=0x7f17d8135970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:11.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17d77fe640 1 --2- 172.21.15.71:0/1810305081 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17d8137830 0x7f17d8135970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:11.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17d77fe640 10 monclient: _finish_hunting 0 2024-04-19T01:42:11.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17d77fe640 1 monclient: found mon.noname-b 2024-04-19T01:42:11.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:11.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17d77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.71:3300/0 2024-04-19T01:42:11.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17d77fe640 1 -- 172.21.15.71:0/1810305081 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17d812bbc0 con 0x7f17d80a0b60 2024-04-19T01:42:11.292 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17d77fe640 1 -- 172.21.15.71:0/1810305081 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17d80a0b60 msgr2=0x7f17d81cffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:11.293 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17d77fe640 1 --2- 172.21.15.71:0/1810305081 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17d80a0b60 0x7f17d81cffc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f17cc0097f0 tx=0x7f17cc0191d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:11.293 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17bffff640 1 -- 172.21.15.71:0/1810305081 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17cc0ba070 con 0x7f17d80a0b60 2024-04-19T01:42:11.293 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:11.293 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17bffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:42:11.293 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17bffff640 10 monclient: dump: 2024-04-19T01:42:11.293 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-04-19T01:42:11.293 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:11.293 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:11.293 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:11.293 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:11.293 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:11.293 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:11.293 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:11.294 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:11.294 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:11.294 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17bffff640 10 monclient: _finish_auth 0 2024-04-19T01:42:11.294 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17bffff640 10 monclient: _check_auth_tickets 2024-04-19T01:42:11.294 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:11.294 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17de10c640 5 monclient: authenticate success, global_id 4126 2024-04-19T01:42:11.294 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17de10c640 10 monclient: _renew_subs 2024-04-19T01:42:11.294 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17bffff640 1 -- 172.21.15.71:0/1810305081 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17cc0b5870 con 0x7f17d80a0b60 2024-04-19T01:42:11.294 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17de10c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:42:11.294 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17de10c640 1 -- 172.21.15.71:0/1810305081 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17d812bea0 con 0x7f17d80a0b60 2024-04-19T01:42:11.294 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17bffff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:11.295 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17bffff640 1 -- 172.21.15.71:0/1810305081 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17cc0b0040 con 0x7f17d80a0b60 2024-04-19T01:42:11.295 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:11.295 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17bffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:42:11.295 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17bffff640 10 monclient: dump: 2024-04-19T01:42:11.295 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-04-19T01:42:11.295 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:11.295 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:11.295 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:11.295 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:11.295 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:11.295 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:11.295 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:11.295 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:11.296 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:11.296 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17de10c640 10 monclient: _renew_subs 2024-04-19T01:42:11.296 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17de10c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:42:11.296 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17de10c640 1 -- 172.21.15.71:0/1810305081 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17d8140380 con 0x7f17d80a0b60 2024-04-19T01:42:11.296 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17d57fa640 4 set_mon_vals no callback set 2024-04-19T01:42:11.296 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17bffff640 1 -- 172.21.15.71:0/1810305081 <== mon.1 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f17cc0bd070 con 0x7f17d80a0b60 2024-04-19T01:42:11.296 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17bffff640 1 -- 172.21.15.71:0/1810305081 <== mon.1 v2:172.21.15.71:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f17cc0bfc50 con 0x7f17d80a0b60 2024-04-19T01:42:11.296 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17de10c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:11.296 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17de10c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:11.296 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17de10c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:42:11.296 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.289+0000 7f17de10c640 1 -- 172.21.15.71:0/1810305081 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17d8068ab0 con 0x7f17d80a0b60 2024-04-19T01:42:11.297 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.293+0000 7f17bffff640 1 -- 172.21.15.71:0/1810305081 <== mon.1 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f17cc0160b0 con 0x7f17d80a0b60 2024-04-19T01:42:11.297 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.293+0000 7f17bffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:11.297 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.293+0000 7f17bffff640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:11.357 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.361+0000 7f17de10c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "230f6e15-548c-4338-a8ec-067b8f85bd2e", "id": 3}] 2024-04-19T01:42:11.357 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.361+0000 7f17de10c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "230f6e15-548c-4338-a8ec-067b8f85bd2e", "id": 3}] 2024-04-19T01:42:11.357 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.361+0000 7f17de10c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:42:11.357 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:11.361+0000 7f17de10c640 1 -- 172.21.15.71:0/1810305081 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd new", "uuid": "230f6e15-548c-4338-a8ec-067b8f85bd2e", "id": 3} v 0) -- 0x7f17d806c070 con 0x7f17d80a0b60 2024-04-19T01:42:11.643 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-04-19T01:42:11.645+0000 7f76bc1b9180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-19T01:42:11.696 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-04-19T01:42:11.701+0000 7f76bc1b9180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-19T01:42:11.748 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-04-19T01:42:11.753+0000 7f76bc1b9180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-19T01:42:11.824 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-04-19T01:42:11.829+0000 7f76bc1b9180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-19T01:42:11.875 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-04-19T01:42:11.877+0000 7f76bc1b9180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-19T01:42:11.950 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-04-19T01:42:11.953+0000 7f76bc1b9180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-19T01:42:12.003 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-04-19T01:42:12.005+0000 7f76bc1b9180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-19T01:42:12.054 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.057+0000 7f17bffff640 1 -- 172.21.15.71:0/1810305081 <== mon.1 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "230f6e15-548c-4338-a8ec-067b8f85bd2e", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f17cc0ab4c0 con 0x7f17d80a0b60 2024-04-19T01:42:12.054 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.057+0000 7f17bffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "230f6e15-548c-4338-a8ec-067b8f85bd2e", "id": 3}] 2024-04-19T01:42:12.054 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.057+0000 7f17bffff640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:12.054 INFO:teuthology.orchestra.run.smithi071.stdout:3 2024-04-19T01:42:12.055 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.057+0000 7f17de10c640 10 monclient: shutdown 2024-04-19T01:42:12.055 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.057+0000 7f17de10c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:12.056 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.057+0000 7f17de10c640 1 -- 172.21.15.71:0/1810305081 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17d80a0b60 msgr2=0x7f17d81cffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:12.056 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.057+0000 7f17de10c640 1 --2- 172.21.15.71:0/1810305081 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17d80a0b60 0x7f17d81cffc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f17cc0097f0 tx=0x7f17cc0191d0 comp rx=0 tx=0).stop 2024-04-19T01:42:12.056 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.057+0000 7f17de10c640 1 -- 172.21.15.71:0/1810305081 shutdown_connections 2024-04-19T01:42:12.056 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.057+0000 7f17de10c640 1 --2- 172.21.15.71:0/1810305081 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f17d80a1650 0x7f17d8135430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:12.056 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.061+0000 7f17de10c640 1 --2- 172.21.15.71:0/1810305081 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f17d8137830 0x7f17d8135970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:12.056 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.061+0000 7f17de10c640 1 --2- 172.21.15.71:0/1810305081 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17d80a0b60 0x7f17d81cffc0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:12.056 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.061+0000 7f17de10c640 1 -- 172.21.15.71:0/1810305081 >> 172.21.15.71:0/1810305081 conn(0x7f17d80a6160 msgr2=0x7f17d809f570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:12.056 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.061+0000 7f17de10c640 1 -- 172.21.15.71:0/1810305081 shutdown_connections 2024-04-19T01:42:12.056 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.061+0000 7f17de10c640 1 -- 172.21.15.71:0/1810305081 wait complete. 2024-04-19T01:42:12.072 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd new 06789be2-244f-4b6e-ada2-30b6771676af 4 2024-04-19T01:42:12.155 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-04-19T01:42:12.161+0000 7f76bc1b9180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-19T01:42:12.185 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a91f9e640 10 monclient: get_monmap_and_config 2024-04-19T01:42:12.185 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a91f9e640 10 monclient: build_initial_monmap 2024-04-19T01:42:12.185 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a91f9e640 10 monclient: monmap: 2024-04-19T01:42:12.185 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-04-19T01:42:12.185 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:12.185 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-04-19T01:42:12.191554+0000 2024-04-19T01:42:12.185 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-04-19T01:42:12.191554+0000 2024-04-19T01:42:12.186 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:12.186 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:12.186 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:12.186 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:12.186 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:12.186 INFO:teuthology.orchestra.run.smithi071.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:12.186 INFO:teuthology.orchestra.run.smithi071.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:12.186 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:12.188 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a91f9e640 1 Processor -- start 2024-04-19T01:42:12.188 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a91f9e640 1 -- start start 2024-04-19T01:42:12.188 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a91f9e640 10 monclient: init 2024-04-19T01:42:12.188 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a91f9e640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:12.189 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a91f9e640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-04-19T01:42:12.189 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a91f9e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4a8c0a2360 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:12.189 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a91f9e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4a8c0a2e50 0x7f4a8c072bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:12.189 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a91f9e640 10 monclient(hunting): picked mon.noname-c con 0x7f4a8c0a2e50 addr v2:172.21.15.71:3300/0 2024-04-19T01:42:12.189 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a91f9e640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4a8c13a6c0 0x7f4a8c13cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:12.189 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a91f9e640 10 monclient(hunting): picked mon.noname-a con 0x7f4a8c13a6c0 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:12.189 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a91f9e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a8c075e00 con 0x7f4a8c0a2360 2024-04-19T01:42:12.189 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a91f9e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:12.189 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a91f9e640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f4a8c072260 con 0x7f4a8c13a6c0 2024-04-19T01:42:12.189 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a91f9e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:12.189 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a91f9e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f4a8c0730f0 con 0x7f4a8c0a2e50 2024-04-19T01:42:12.189 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a91f9e640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:12.189 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a91f9e640 10 monclient(hunting): authenticate will time out at 917.655464s 2024-04-19T01:42:12.189 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a83fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4a8c0a2e50 0x7f4a8c072bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:12.189 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a83fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4a8c0a2e50 0x7f4a8c072bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52136/0 (socket says 172.21.15.71:52136) 2024-04-19T01:42:12.190 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a83fff640 1 -- 172.21.15.71:0/748220860 learned_addr learned my addr 172.21.15.71:0/748220860 (peer_addr_for_me v2:172.21.15.71:0/0) 2024-04-19T01:42:12.190 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a83fff640 10 monclient(hunting): get_auth_request con 0x7f4a8c0a2e50 auth_method 0 2024-04-19T01:42:12.190 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:12.190 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a83fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:12.190 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a83fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:12.190 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:12.190 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:12.190 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:12.190 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a8bfff640 1 --2- 172.21.15.71:0/748220860 >> v2:172.21.15.62:3300/0 conn(0x7f4a8c13a6c0 0x7f4a8c13cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:12.190 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a8bfff640 10 monclient(hunting): get_auth_request con 0x7f4a8c13a6c0 auth_method 0 2024-04-19T01:42:12.190 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:12.190 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a8bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:12.190 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:12.190 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.189+0000 7f4a83fff640 10 monclient(hunting): handle_auth_done global_id 4129 payload 274 2024-04-19T01:42:12.191 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.193+0000 7f4a83fff640 1 -- 172.21.15.71:0/748220860 >> v2:172.21.15.62:3300/0 conn(0x7f4a8c13a6c0 msgr2=0x7f4a8c13cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:12.191 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.193+0000 7f4a83fff640 1 --2- 172.21.15.71:0/748220860 >> v2:172.21.15.62:3300/0 conn(0x7f4a8c13a6c0 0x7f4a8c13cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:12.191 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.193+0000 7f4a83fff640 1 -- 172.21.15.71:0/748220860 >> v1:172.21.15.62:6789/0 conn(0x7f4a8c0a2360 legacy=0x7f4a8c0a2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:12.191 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.193+0000 7f4a83fff640 10 monclient: _finish_hunting 0 2024-04-19T01:42:12.191 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.193+0000 7f4a83fff640 1 monclient: found mon.noname-c 2024-04-19T01:42:12.191 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.193+0000 7f4a83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:12.191 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.193+0000 7f4a83fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-04-19T01:42:12.191 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.193+0000 7f4a83fff640 1 -- 172.21.15.71:0/748220860 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a8c0759c0 con 0x7f4a8c0a2e50 2024-04-19T01:42:12.192 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a83fff640 1 -- 172.21.15.71:0/748220860 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a8c0a2e50 msgr2=0x7f4a8c072bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:12.193 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a83fff640 1 --2- 172.21.15.71:0/748220860 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a8c0a2e50 0x7f4a8c072bb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c06ea70 tx=0x7f4a7401c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=faedb8852e277a9e server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:12.193 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8affd640 1 -- 172.21.15.71:0/748220860 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a740b1070 con 0x7f4a8c0a2e50 2024-04-19T01:42:12.193 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:12.193 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:12.193 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8affd640 10 monclient: dump: 2024-04-19T01:42:12.193 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-04-19T01:42:12.193 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:12.193 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:12.193 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:12.193 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:12.193 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:12.193 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:12.193 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:12.194 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:12.194 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:12.194 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8affd640 10 monclient: _finish_auth 0 2024-04-19T01:42:12.194 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8affd640 10 monclient: _check_auth_tickets 2024-04-19T01:42:12.194 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:12.194 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8affd640 1 -- 172.21.15.71:0/748220860 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a740ac620 con 0x7f4a8c0a2e50 2024-04-19T01:42:12.194 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8affd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:12.194 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8affd640 1 -- 172.21.15.71:0/748220860 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a740ac9f0 con 0x7f4a8c0a2e50 2024-04-19T01:42:12.194 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 5 monclient: authenticate success, global_id 4129 2024-04-19T01:42:12.194 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:12.194 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 4 set_mon_vals no callback set 2024-04-19T01:42:12.194 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:12.194 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:42:12.194 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8affd640 10 monclient: dump: 2024-04-19T01:42:12.194 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-04-19T01:42:12.195 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:12.195 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:12.195 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:12.195 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:12.195 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:12.195 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:12.195 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:12.195 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:12.195 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:12.195 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 10 monclient: shutdown 2024-04-19T01:42:12.195 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:12.195 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 1 -- 172.21.15.71:0/748220860 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a8c0a2e50 msgr2=0x7f4a8c072bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:12.195 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 1 --2- 172.21.15.71:0/748220860 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a8c0a2e50 0x7f4a8c072bb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c06ea70 tx=0x7f4a7401c3e0 comp rx=0 tx=0).stop 2024-04-19T01:42:12.195 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 1 -- 172.21.15.71:0/748220860 shutdown_connections 2024-04-19T01:42:12.195 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 1 --2- 172.21.15.71:0/748220860 >> v2:172.21.15.62:3300/0 conn(0x7f4a8c13a6c0 0x7f4a8c13cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:12.195 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 1 --2- 172.21.15.71:0/748220860 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a8c0a2e50 0x7f4a8c072bb0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:12.196 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 1 -- 172.21.15.71:0/748220860 >> 172.21.15.71:0/748220860 conn(0x7f4a8c0a7070 msgr2=0x7f4a8c0a3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:12.196 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 1 -- 172.21.15.71:0/748220860 shutdown_connections 2024-04-19T01:42:12.196 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 1 -- 172.21.15.71:0/748220860 wait complete. 2024-04-19T01:42:12.200 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 10 monclient: build_initial_monmap 2024-04-19T01:42:12.200 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 10 monclient: monmap: 2024-04-19T01:42:12.200 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-04-19T01:42:12.200 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:12.200 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 0.000000 2024-04-19T01:42:12.200 INFO:teuthology.orchestra.run.smithi071.stderr:created 0.000000 2024-04-19T01:42:12.200 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:12.200 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:12.200 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:12.201 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:12.201 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:12.201 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:12.201 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 1 Processor -- start 2024-04-19T01:42:12.201 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 1 -- start start 2024-04-19T01:42:12.201 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 10 monclient: init 2024-04-19T01:42:12.201 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:12.201 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-19T01:42:12.201 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a8c0a2360 0x7f4a8c09fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:12.201 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 10 monclient(hunting): picked mon.noname-b con 0x7f4a8c0a2360 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:12.201 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a8c0a2e50 0x7f4a8c11d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:12.202 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 10 monclient(hunting): picked mon.noname-c con 0x7f4a8c0a2e50 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:12.202 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a8c13a6c0 0x7f4a8c1ccfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:12.202 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 10 monclient(hunting): picked mon.noname-a con 0x7f4a8c13a6c0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:12.202 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:12.202 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8c13fba0 con 0x7f4a8c13a6c0 2024-04-19T01:42:12.202 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:12.202 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8c13fa20 con 0x7f4a8c0a2360 2024-04-19T01:42:12.202 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:12.202 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f4a8c13fd20 con 0x7f4a8c0a2e50 2024-04-19T01:42:12.202 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:12.202 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a91f9e640 10 monclient(hunting): authenticate will time out at 917.662969s 2024-04-19T01:42:12.202 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8b7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a8c0a2360 0x7f4a8c09fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:12.202 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8b7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a8c0a2360 0x7f4a8c09fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52144/0 (socket says 172.21.15.71:52144) 2024-04-19T01:42:12.202 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8b7fe640 1 -- 172.21.15.71:0/1035375331 learned_addr learned my addr 172.21.15.71:0/1035375331 (peer_addr_for_me v2:172.21.15.71:0/0) 2024-04-19T01:42:12.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8b7fe640 10 monclient(hunting): get_auth_request con 0x7f4a8c0a2360 auth_method 0 2024-04-19T01:42:12.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:12.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:12.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:12.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:12.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:12.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:12.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8bfff640 1 --2- 172.21.15.71:0/1035375331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a8c13a6c0 0x7f4a8c1ccfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:12.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a83fff640 1 --2- 172.21.15.71:0/1035375331 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a8c0a2e50 0x7f4a8c11d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:12.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8bfff640 10 monclient(hunting): get_auth_request con 0x7f4a8c13a6c0 auth_method 0 2024-04-19T01:42:12.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:12.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:12.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:12.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a83fff640 10 monclient(hunting): get_auth_request con 0x7f4a8c0a2e50 auth_method 0 2024-04-19T01:42:12.203 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:12.204 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a83fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:12.204 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a83fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:12.204 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8b7fe640 10 monclient(hunting): handle_auth_done global_id 4132 payload 824 2024-04-19T01:42:12.204 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8b7fe640 1 -- 172.21.15.71:0/1035375331 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a8c0a2e50 msgr2=0x7f4a8c11d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:12.204 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8b7fe640 1 --2- 172.21.15.71:0/1035375331 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a8c0a2e50 0x7f4a8c11d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:12.204 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8b7fe640 1 -- 172.21.15.71:0/1035375331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a8c13a6c0 msgr2=0x7f4a8c1ccfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:12.204 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8b7fe640 1 --2- 172.21.15.71:0/1035375331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a8c13a6c0 0x7f4a8c1ccfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:12.204 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8b7fe640 10 monclient: _finish_hunting 0 2024-04-19T01:42:12.204 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8b7fe640 1 monclient: found mon.noname-b 2024-04-19T01:42:12.204 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:12.204 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.71:3300/0 2024-04-19T01:42:12.204 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.197+0000 7f4a8b7fe640 1 -- 172.21.15.71:0/1035375331 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a8c0a0670 con 0x7f4a8c0a2360 2024-04-19T01:42:12.204 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.201+0000 7f4a8b7fe640 1 -- 172.21.15.71:0/1035375331 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a8c0a2360 msgr2=0x7f4a8c09fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:12.204 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.201+0000 7f4a8b7fe640 1 --2- 172.21.15.71:0/1035375331 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a8c0a2360 0x7f4a8c09fa30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c00c450 tx=0x7f4a7c00c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:12.205 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.201+0000 7f4a88ff9640 1 -- 172.21.15.71:0/1035375331 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c012a40 con 0x7f4a8c0a2360 2024-04-19T01:42:12.205 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.201+0000 7f4a88ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:12.205 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.201+0000 7f4a88ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:42:12.205 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.201+0000 7f4a88ff9640 10 monclient: dump: 2024-04-19T01:42:12.205 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-04-19T01:42:12.205 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:12.205 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:12.205 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:12.205 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:12.205 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:12.205 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:12.205 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:12.205 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:12.205 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:12.206 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.201+0000 7f4a88ff9640 10 monclient: _finish_auth 0 2024-04-19T01:42:12.206 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.201+0000 7f4a88ff9640 10 monclient: _check_auth_tickets 2024-04-19T01:42:12.206 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.201+0000 7f4a88ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:12.206 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.201+0000 7f4a88ff9640 1 -- 172.21.15.71:0/1035375331 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a7c017070 con 0x7f4a8c0a2360 2024-04-19T01:42:12.206 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.201+0000 7f4a91f9e640 5 monclient: authenticate success, global_id 4132 2024-04-19T01:42:12.206 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.201+0000 7f4a91f9e640 10 monclient: _renew_subs 2024-04-19T01:42:12.206 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.201+0000 7f4a91f9e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:42:12.206 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.201+0000 7f4a91f9e640 1 -- 172.21.15.71:0/1035375331 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a8c11f580 con 0x7f4a8c0a2360 2024-04-19T01:42:12.206 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.201+0000 7f4a88ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:12.206 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.201+0000 7f4a88ff9640 1 -- 172.21.15.71:0/1035375331 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c012e40 con 0x7f4a8c0a2360 2024-04-19T01:42:12.206 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.201+0000 7f4a88ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:12.206 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.201+0000 7f4a88ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:42:12.206 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.201+0000 7f4a88ff9640 10 monclient: dump: 2024-04-19T01:42:12.206 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-04-19T01:42:12.207 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:12.207 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:12.207 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:12.207 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:12.207 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:12.207 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:12.207 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:12.207 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:12.207 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:12.207 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.201+0000 7f4a8a7fc640 4 set_mon_vals no callback set 2024-04-19T01:42:12.207 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.201+0000 7f4a91f9e640 10 monclient: _renew_subs 2024-04-19T01:42:12.207 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.201+0000 7f4a91f9e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:42:12.207 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.201+0000 7f4a91f9e640 1 -- 172.21.15.71:0/1035375331 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a8c1357d0 con 0x7f4a8c0a2360 2024-04-19T01:42:12.207 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.201+0000 7f4a88ff9640 1 -- 172.21.15.71:0/1035375331 <== mon.1 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4a7c01c460 con 0x7f4a8c0a2360 2024-04-19T01:42:12.208 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.201+0000 7f4a88ff9640 1 -- 172.21.15.71:0/1035375331 <== mon.1 v2:172.21.15.71:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f4a7c011380 con 0x7f4a8c0a2360 2024-04-19T01:42:12.208 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.201+0000 7f4a91f9e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:12.208 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.201+0000 7f4a91f9e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:12.208 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.201+0000 7f4a91f9e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:42:12.208 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.201+0000 7f4a91f9e640 1 -- 172.21.15.71:0/1035375331 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a8c068ab0 con 0x7f4a8c0a2360 2024-04-19T01:42:12.208 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.205+0000 7f4a88ff9640 1 -- 172.21.15.71:0/1035375331 <== mon.1 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f4a7c01b470 con 0x7f4a8c0a2360 2024-04-19T01:42:12.208 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.205+0000 7f4a88ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:12.208 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.205+0000 7f4a88ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:12.221 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-04-19T01:42:12.225+0000 7f76bc1b9180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-19T01:42:12.269 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.273+0000 7f4a91f9e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "06789be2-244f-4b6e-ada2-30b6771676af", "id": 4}] 2024-04-19T01:42:12.270 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.273+0000 7f4a91f9e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "06789be2-244f-4b6e-ada2-30b6771676af", "id": 4}] 2024-04-19T01:42:12.270 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.273+0000 7f4a91f9e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:42:12.270 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.273+0000 7f4a91f9e640 1 -- 172.21.15.71:0/1035375331 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd new", "uuid": "06789be2-244f-4b6e-ada2-30b6771676af", "id": 4} v 0) -- 0x7f4a8c11fa80 con 0x7f4a8c0a2360 2024-04-19T01:42:12.536 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.541+0000 7f4a88ff9640 1 -- 172.21.15.71:0/1035375331 <== mon.1 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "06789be2-244f-4b6e-ada2-30b6771676af", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f4a7c01ca00 con 0x7f4a8c0a2360 2024-04-19T01:42:12.536 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.541+0000 7f4a88ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "06789be2-244f-4b6e-ada2-30b6771676af", "id": 4}] 2024-04-19T01:42:12.536 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.541+0000 7f4a88ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:12.536 INFO:teuthology.orchestra.run.smithi071.stdout:4 2024-04-19T01:42:12.538 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.541+0000 7f4a91f9e640 10 monclient: shutdown 2024-04-19T01:42:12.538 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.541+0000 7f4a91f9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:12.539 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.541+0000 7f4a91f9e640 1 -- 172.21.15.71:0/1035375331 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a8c0a2360 msgr2=0x7f4a8c09fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:12.539 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.541+0000 7f4a91f9e640 1 --2- 172.21.15.71:0/1035375331 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a8c0a2360 0x7f4a8c09fa30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c00c450 tx=0x7f4a7c00c910 comp rx=0 tx=0).stop 2024-04-19T01:42:12.539 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.541+0000 7f4a91f9e640 1 -- 172.21.15.71:0/1035375331 shutdown_connections 2024-04-19T01:42:12.539 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.541+0000 7f4a91f9e640 1 --2- 172.21.15.71:0/1035375331 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a8c0a2e50 0x7f4a8c11d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:12.539 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.541+0000 7f4a91f9e640 1 --2- 172.21.15.71:0/1035375331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a8c13a6c0 0x7f4a8c1ccfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:12.539 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.541+0000 7f4a91f9e640 1 --2- 172.21.15.71:0/1035375331 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a8c0a2360 0x7f4a8c09fa30 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:12.539 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.541+0000 7f4a91f9e640 1 -- 172.21.15.71:0/1035375331 >> 172.21.15.71:0/1035375331 conn(0x7f4a8c0a7070 msgr2=0x7f4a8c0a4800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:12.539 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.541+0000 7f4a91f9e640 1 -- 172.21.15.71:0/1035375331 shutdown_connections 2024-04-19T01:42:12.539 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.541+0000 7f4a91f9e640 1 -- 172.21.15.71:0/1035375331 wait complete. 2024-04-19T01:42:12.565 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd new 552be339-732f-4801-b14d-3d9b08e7f1f3 5 2024-04-19T01:42:12.676 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cdc529640 10 monclient: get_monmap_and_config 2024-04-19T01:42:12.676 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cdc529640 10 monclient: build_initial_monmap 2024-04-19T01:42:12.676 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cdc529640 10 monclient: monmap: 2024-04-19T01:42:12.676 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-04-19T01:42:12.676 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:12.676 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-04-19T01:42:12.678158+0000 2024-04-19T01:42:12.676 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-04-19T01:42:12.678158+0000 2024-04-19T01:42:12.676 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:12.676 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:12.676 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:12.676 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:12.676 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:12.677 INFO:teuthology.orchestra.run.smithi071.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:12.677 INFO:teuthology.orchestra.run.smithi071.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:12.677 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:12.677 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cdc529640 1 Processor -- start 2024-04-19T01:42:12.677 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cdc529640 1 -- start start 2024-04-19T01:42:12.677 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cdc529640 10 monclient: init 2024-04-19T01:42:12.677 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cdc529640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:12.677 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cdc529640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-04-19T01:42:12.677 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cdc529640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3cd40a3470 0x7f3cd4072a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:12.677 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cdc529640 10 monclient(hunting): picked mon.noname-c con 0x7f3cd40a3470 addr v2:172.21.15.71:3300/0 2024-04-19T01:42:12.677 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cdc529640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3cd40a3850 addr v1:172.21.15.71:6789/0 2024-04-19T01:42:12.677 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cdc529640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3cd413a670 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:12.677 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cdc529640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cd4071770 con 0x7f3cd413a670 2024-04-19T01:42:12.677 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cdc529640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cd4070010 con 0x7f3cd40a3850 2024-04-19T01:42:12.677 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cdc529640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:12.678 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cdc529640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f3cd4072f50 con 0x7f3cd40a3470 2024-04-19T01:42:12.678 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cdc529640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:12.678 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cdc529640 10 monclient(hunting): authenticate will time out at 918.141769s 2024-04-19T01:42:12.678 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cda2c5640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3cd40a3470 0x7f3cd4072a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:12.678 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cd9ac4640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f3cd40a3850 0x7f3cd40a3c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:47934/0 (socket says 172.21.15.71:47934) 2024-04-19T01:42:12.678 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cd9ac4640 1 -- 172.21.15.71:0/2363137981 learned_addr learned my addr 172.21.15.71:0/2363137981 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-04-19T01:42:12.678 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cda2c5640 10 monclient(hunting): get_auth_request con 0x7f3cd40a3470 auth_method 0 2024-04-19T01:42:12.678 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cda2c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:12.678 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cda2c5640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:12.678 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cda2c5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:12.678 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cda2c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:12.678 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cda2c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:12.678 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cda2c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:12.678 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cda2c5640 10 monclient(hunting): handle_auth_done global_id 4135 payload 274 2024-04-19T01:42:12.679 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cda2c5640 1 -- 172.21.15.71:0/2363137981 >> v1:172.21.15.71:6789/0 conn(0x7f3cd40a3850 legacy=0x7f3cd40a3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:12.679 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cda2c5640 1 -- 172.21.15.71:0/2363137981 >> v1:172.21.15.62:6789/0 conn(0x7f3cd413a670 legacy=0x7f3cd413ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:12.679 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cda2c5640 10 monclient: _finish_hunting 0 2024-04-19T01:42:12.679 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cda2c5640 1 monclient: found mon.noname-c 2024-04-19T01:42:12.679 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cda2c5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:12.679 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cda2c5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-04-19T01:42:12.679 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.677+0000 7f3cda2c5640 1 -- 172.21.15.71:0/2363137981 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cd40a52e0 con 0x7f3cd40a3470 2024-04-19T01:42:12.681 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.681+0000 7f3cda2c5640 1 -- 172.21.15.71:0/2363137981 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cd40a3470 msgr2=0x7f3cd4072a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:12.681 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.681+0000 7f3cda2c5640 1 --2- 172.21.15.71:0/2363137981 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cd40a3470 0x7f3cd4072a10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4009d90 tx=0x7f3cc401c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=711b6248b5373eb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:12.681 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd92c3640 1 -- 172.21.15.71:0/2363137981 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc40a6040 con 0x7f3cd40a3470 2024-04-19T01:42:12.681 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd92c3640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:12.681 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd92c3640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:12.682 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd92c3640 10 monclient: dump: 2024-04-19T01:42:12.682 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-04-19T01:42:12.682 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:12.682 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:12.682 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:12.682 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:12.682 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:12.682 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:12.682 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:12.682 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:12.682 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:12.682 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd92c3640 10 monclient: _finish_auth 0 2024-04-19T01:42:12.682 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd92c3640 10 monclient: _check_auth_tickets 2024-04-19T01:42:12.682 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd92c3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:12.683 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd92c3640 1 -- 172.21.15.71:0/2363137981 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cc40af070 con 0x7f3cd40a3470 2024-04-19T01:42:12.683 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 5 monclient: authenticate success, global_id 4135 2024-04-19T01:42:12.683 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:42:12.683 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd92c3640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:12.683 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:12.683 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd92c3640 1 -- 172.21.15.71:0/2363137981 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc40aa940 con 0x7f3cd40a3470 2024-04-19T01:42:12.683 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 4 set_mon_vals no callback set 2024-04-19T01:42:12.683 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd92c3640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:12.683 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd92c3640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:42:12.683 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd92c3640 10 monclient: dump: 2024-04-19T01:42:12.683 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-04-19T01:42:12.683 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:12.683 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:12.683 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:12.683 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:12.684 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:12.684 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:12.684 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:12.684 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:12.684 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:12.684 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 10 monclient: shutdown 2024-04-19T01:42:12.684 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:12.684 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 1 -- 172.21.15.71:0/2363137981 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cd40a3470 msgr2=0x7f3cd4072a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:12.684 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 1 --2- 172.21.15.71:0/2363137981 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cd40a3470 0x7f3cd4072a10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4009d90 tx=0x7f3cc401c650 comp rx=0 tx=0).stop 2024-04-19T01:42:12.684 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 1 -- 172.21.15.71:0/2363137981 shutdown_connections 2024-04-19T01:42:12.684 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 1 --2- 172.21.15.71:0/2363137981 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cd40a3470 0x7f3cd4072a10 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:12.684 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 1 -- 172.21.15.71:0/2363137981 >> 172.21.15.71:0/2363137981 conn(0x7f3cd40a4560 msgr2=0x7f3cd40a4970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:12.684 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 1 -- 172.21.15.71:0/2363137981 shutdown_connections 2024-04-19T01:42:12.684 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 1 -- 172.21.15.71:0/2363137981 wait complete. 2024-04-19T01:42:12.684 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 10 monclient: build_initial_monmap 2024-04-19T01:42:12.685 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 10 monclient: monmap: 2024-04-19T01:42:12.685 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-04-19T01:42:12.685 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:12.685 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 0.000000 2024-04-19T01:42:12.685 INFO:teuthology.orchestra.run.smithi071.stderr:created 0.000000 2024-04-19T01:42:12.685 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:12.685 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:12.685 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:12.685 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:12.685 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:12.685 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:12.685 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 1 Processor -- start 2024-04-19T01:42:12.685 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 1 -- start start 2024-04-19T01:42:12.685 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 10 monclient: init 2024-04-19T01:42:12.689 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:12.689 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-19T01:42:12.689 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3cd40a3470 0x7f3cd41cd170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:12.689 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 10 monclient(hunting): picked mon.noname-c con 0x7f3cd40a3470 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:12.689 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cd40a3850 0x7f3cd4134310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:12.689 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 10 monclient(hunting): picked mon.noname-b con 0x7f3cd40a3850 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:12.689 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cd413a670 0x7f3cd409d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:12.689 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 10 monclient(hunting): picked mon.noname-a con 0x7f3cd413a670 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:12.690 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:12.690 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd4140bb0 con 0x7f3cd413a670 2024-04-19T01:42:12.690 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:12.690 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd40704d0 con 0x7f3cd40a3850 2024-04-19T01:42:12.690 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:12.690 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f3cd4140d30 con 0x7f3cd40a3470 2024-04-19T01:42:12.690 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:12.690 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdc529640 10 monclient(hunting): authenticate will time out at 918.150763s 2024-04-19T01:42:12.690 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd9ac4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cd40a3850 0x7f3cd4134310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:12.690 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd9ac4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cd40a3850 0x7f3cd4134310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52162/0 (socket says 172.21.15.71:52162) 2024-04-19T01:42:12.690 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd9ac4640 1 -- 172.21.15.71:0/505921501 learned_addr learned my addr 172.21.15.71:0/505921501 (peer_addr_for_me v2:172.21.15.71:0/0) 2024-04-19T01:42:12.690 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd9ac4640 10 monclient(hunting): get_auth_request con 0x7f3cd40a3850 auth_method 0 2024-04-19T01:42:12.690 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd9ac4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:12.690 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd9ac4640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:12.691 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd9ac4640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:12.691 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd9ac4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:12.691 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd9ac4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:12.691 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd9ac4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:12.691 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd9ac4640 10 monclient(hunting): handle_auth_done global_id 4138 payload 824 2024-04-19T01:42:12.691 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cda2c5640 1 --2- 172.21.15.71:0/505921501 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3cd40a3470 0x7f3cd41cd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:12.691 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd9ac4640 1 -- 172.21.15.71:0/505921501 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3cd40a3470 msgr2=0x7f3cd41cd170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:12.691 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cdaac6640 1 --2- 172.21.15.71:0/505921501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cd413a670 0x7f3cd409d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:12.691 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd9ac4640 1 --2- 172.21.15.71:0/505921501 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3cd40a3470 0x7f3cd41cd170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:12.691 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd9ac4640 1 -- 172.21.15.71:0/505921501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cd413a670 msgr2=0x7f3cd409d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:12.691 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd9ac4640 1 --2- 172.21.15.71:0/505921501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cd413a670 0x7f3cd409d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:12.691 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd9ac4640 10 monclient: _finish_hunting 0 2024-04-19T01:42:12.691 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd9ac4640 1 monclient: found mon.noname-b 2024-04-19T01:42:12.691 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd9ac4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:12.692 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd9ac4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.71:3300/0 2024-04-19T01:42:12.692 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.685+0000 7f3cd9ac4640 1 -- 172.21.15.71:0/505921501 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cd409f400 con 0x7f3cd40a3850 2024-04-19T01:42:12.692 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.689+0000 7f3cd9ac4640 1 -- 172.21.15.71:0/505921501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cd40a3850 msgr2=0x7f3cd4134310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:12.692 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.689+0000 7f3cd9ac4640 1 --2- 172.21.15.71:0/505921501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cd40a3850 0x7f3cd4134310 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3cc800c4a0 tx=0x7f3cc800c960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:12.692 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.689+0000 7f3cd2ffd640 1 -- 172.21.15.71:0/505921501 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc8017070 con 0x7f3cd40a3850 2024-04-19T01:42:12.692 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.689+0000 7f3cd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:12.692 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.689+0000 7f3cd2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:42:12.692 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.689+0000 7f3cd2ffd640 10 monclient: dump: 2024-04-19T01:42:12.692 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-04-19T01:42:12.692 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:12.692 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:12.692 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:12.692 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:12.692 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:12.692 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:12.693 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:12.693 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:12.693 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:12.693 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.689+0000 7f3cd2ffd640 10 monclient: _finish_auth 0 2024-04-19T01:42:12.693 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.689+0000 7f3cd2ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:42:12.693 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.689+0000 7f3cd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:12.693 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.689+0000 7f3cd2ffd640 1 -- 172.21.15.71:0/505921501 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cc8007070 con 0x7f3cd40a3850 2024-04-19T01:42:12.693 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.689+0000 7f3cd2ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:12.693 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.689+0000 7f3cd2ffd640 1 -- 172.21.15.71:0/505921501 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc8007460 con 0x7f3cd40a3850 2024-04-19T01:42:12.693 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.689+0000 7f3cd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:12.693 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.689+0000 7f3cd2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:42:12.693 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.689+0000 7f3cd2ffd640 10 monclient: dump: 2024-04-19T01:42:12.693 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-04-19T01:42:12.694 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:12.694 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:12.694 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:12.694 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:12.694 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-04-19T01:42:12.694 INFO:teuthology.orchestra.run.smithi071.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:12.694 INFO:teuthology.orchestra.run.smithi071.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:12.694 INFO:teuthology.orchestra.run.smithi071.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:12.694 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:42:12.694 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.689+0000 7f3cd8ac2640 4 set_mon_vals no callback set 2024-04-19T01:42:12.694 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.689+0000 7f3cdc529640 5 monclient: authenticate success, global_id 4138 2024-04-19T01:42:12.694 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.689+0000 7f3cdc529640 10 monclient: _renew_subs 2024-04-19T01:42:12.694 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.689+0000 7f3cdc529640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:42:12.695 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.689+0000 7f3cdc529640 1 -- 172.21.15.71:0/505921501 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cd40a0e50 con 0x7f3cd40a3850 2024-04-19T01:42:12.695 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.689+0000 7f3cdc529640 10 monclient: _renew_subs 2024-04-19T01:42:12.695 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.689+0000 7f3cdc529640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:42:12.695 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.689+0000 7f3cdc529640 1 -- 172.21.15.71:0/505921501 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cd40a0b20 con 0x7f3cd40a3850 2024-04-19T01:42:12.695 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.689+0000 7f3cd2ffd640 1 -- 172.21.15.71:0/505921501 <== mon.1 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3cc801a070 con 0x7f3cd40a3850 2024-04-19T01:42:12.695 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.689+0000 7f3cd2ffd640 1 -- 172.21.15.71:0/505921501 <== mon.1 v2:172.21.15.71:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f3cc8012a10 con 0x7f3cd40a3850 2024-04-19T01:42:12.695 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.689+0000 7f3cdc529640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:12.695 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.689+0000 7f3cdc529640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:12.695 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.689+0000 7f3cdc529640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:42:12.695 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.689+0000 7f3cdc529640 1 -- 172.21.15.71:0/505921501 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cd4068ab0 con 0x7f3cd40a3850 2024-04-19T01:42:12.695 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.693+0000 7f3cd2ffd640 1 -- 172.21.15.71:0/505921501 <== mon.1 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f3cc8021020 con 0x7f3cd40a3850 2024-04-19T01:42:12.695 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.693+0000 7f3cd2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:12.695 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.693+0000 7f3cd2ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:12.756 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.761+0000 7f3cdc529640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "552be339-732f-4801-b14d-3d9b08e7f1f3", "id": 5}] 2024-04-19T01:42:12.756 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.761+0000 7f3cdc529640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "552be339-732f-4801-b14d-3d9b08e7f1f3", "id": 5}] 2024-04-19T01:42:12.756 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.761+0000 7f3cdc529640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:42:12.756 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.761+0000 7f3cdc529640 1 -- 172.21.15.71:0/505921501 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd new", "uuid": "552be339-732f-4801-b14d-3d9b08e7f1f3", "id": 5} v 0) -- 0x7f3cd406db90 con 0x7f3cd40a3850 2024-04-19T01:42:12.764 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.769+0000 7f3cd2ffd640 1 -- 172.21.15.71:0/505921501 <== mon.1 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "552be339-732f-4801-b14d-3d9b08e7f1f3", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f3cc8012520 con 0x7f3cd40a3850 2024-04-19T01:42:12.764 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.769+0000 7f3cd2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "552be339-732f-4801-b14d-3d9b08e7f1f3", "id": 5}] 2024-04-19T01:42:12.764 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.769+0000 7f3cd2ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:12.764 INFO:teuthology.orchestra.run.smithi071.stdout:5 2024-04-19T01:42:12.766 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.769+0000 7f3cdc529640 10 monclient: shutdown 2024-04-19T01:42:12.766 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.769+0000 7f3cdc529640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:12.766 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.769+0000 7f3cdc529640 1 -- 172.21.15.71:0/505921501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cd40a3850 msgr2=0x7f3cd4134310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:12.766 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.769+0000 7f3cdc529640 1 --2- 172.21.15.71:0/505921501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cd40a3850 0x7f3cd4134310 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3cc800c4a0 tx=0x7f3cc800c960 comp rx=0 tx=0).stop 2024-04-19T01:42:12.766 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.769+0000 7f3cdc529640 1 -- 172.21.15.71:0/505921501 shutdown_connections 2024-04-19T01:42:12.766 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.769+0000 7f3cdc529640 1 --2- 172.21.15.71:0/505921501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cd413a670 0x7f3cd409d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:12.766 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.769+0000 7f3cdc529640 1 --2- 172.21.15.71:0/505921501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cd40a3850 0x7f3cd4134310 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:12.766 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.769+0000 7f3cdc529640 1 --2- 172.21.15.71:0/505921501 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3cd40a3470 0x7f3cd41cd170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:12.766 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.769+0000 7f3cdc529640 1 -- 172.21.15.71:0/505921501 >> 172.21.15.71:0/505921501 conn(0x7f3cd40a4560 msgr2=0x7f3cd413af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:12.766 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.769+0000 7f3cdc529640 1 -- 172.21.15.71:0/505921501 shutdown_connections 2024-04-19T01:42:12.766 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-19T01:42:12.769+0000 7f3cdc529640 1 -- 172.21.15.71:0/505921501 wait complete. 2024-04-19T01:42:12.780 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-19T01:42:12.781 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-19T01:42:12.783 INFO:tasks.ceph.osd.0:Started 2024-04-19T01:42:12.783 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-19T01:42:12.783 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-19T01:42:12.786 INFO:tasks.ceph.osd.1:Started 2024-04-19T01:42:12.786 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-19T01:42:12.786 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-19T01:42:12.788 INFO:tasks.ceph.osd.2:Started 2024-04-19T01:42:12.788 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-19T01:42:12.788 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-19T01:42:12.791 INFO:tasks.ceph.osd.3:Started 2024-04-19T01:42:12.791 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-19T01:42:12.791 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-19T01:42:12.793 INFO:tasks.ceph.osd.4:Started 2024-04-19T01:42:12.793 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-19T01:42:12.793 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-19T01:42:12.795 INFO:tasks.ceph.osd.5:Started 2024-04-19T01:42:12.796 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-19T01:42:12.805 INFO:tasks.ceph.mgr.x.smithi071.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-19T01:42:12.805 INFO:tasks.ceph.mgr.x.smithi071.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-19T01:42:12.805 INFO:tasks.ceph.mgr.x.smithi071.stderr: from numpy import show_config as show_numpy_config 2024-04-19T01:42:12.859 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-04-19T01:42:12.865+0000 7f76bc1b9180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-19T01:42:12.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f94551dd640 10 monclient: get_monmap_and_config 2024-04-19T01:42:12.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f94551dd640 10 monclient: build_initial_monmap 2024-04-19T01:42:12.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f94551dd640 10 monclient: monmap: 2024-04-19T01:42:12.918 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:12.918 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:12.918 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:12.922125+0000 2024-04-19T01:42:12.918 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:12.922125+0000 2024-04-19T01:42:12.919 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:12.919 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:12.919 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:12.919 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:12.919 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:12.919 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:12.919 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:12.919 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:12.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f94551dd640 1 Processor -- start 2024-04-19T01:42:12.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f94551dd640 1 -- start start 2024-04-19T01:42:12.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f94551dd640 10 monclient: init 2024-04-19T01:42:12.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f94551dd640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:12.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f94551dd640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-04-19T01:42:12.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f94551dd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9450137780 0x7f9450137b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:12.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f94551dd640 10 monclient(hunting): picked mon.noname-b con 0x7f9450137780 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:12.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f94551dd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f94501380a0 0x7f94500a45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:12.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f94551dd640 10 monclient(hunting): picked mon.noname-a con 0x7f94501380a0 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:12.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f94551dd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f94500a60b0 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:12.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f94551dd640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f945009c2e0 con 0x7f94500a60b0 2024-04-19T01:42:12.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f94551dd640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:12.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f94551dd640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f945009bd20 con 0x7f94501380a0 2024-04-19T01:42:12.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f94551dd640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:12.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f94551dd640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f945009c000 con 0x7f9450137780 2024-04-19T01:42:12.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f94551dd640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:12.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f94551dd640 10 monclient(hunting): authenticate will time out at 918.741243s 2024-04-19T01:42:12.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f944ed9d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9450137780 0x7f9450137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:12.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f944e59c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f94501380a0 0x7f94500a45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:12.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f944f59e640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f94500a60b0 0x7f94500a4b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:45424/0 (socket says 172.21.15.62:45424) 2024-04-19T01:42:12.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f944f59e640 1 -- 172.21.15.62:0/759196514 learned_addr learned my addr 172.21.15.62:0/759196514 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:42:12.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f944ed9d640 10 monclient(hunting): get_auth_request con 0x7f9450137780 auth_method 0 2024-04-19T01:42:12.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f944ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:12.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f944ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:12.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f944ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:12.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f944e59c640 10 monclient(hunting): get_auth_request con 0x7f94501380a0 auth_method 0 2024-04-19T01:42:12.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f944e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:12.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f944e59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:12.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f944e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:12.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f944ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:12.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f944ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:12.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f944ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:12.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f944e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:12.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f944e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:12.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f944e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:12.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f944ed9d640 10 monclient(hunting): handle_auth_done global_id 4142 payload 274 2024-04-19T01:42:12.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f944ed9d640 1 -- 172.21.15.62:0/759196514 >> v2:172.21.15.62:3300/0 conn(0x7f94501380a0 msgr2=0x7f94500a45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:12.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f944ed9d640 1 --2- 172.21.15.62:0/759196514 >> v2:172.21.15.62:3300/0 conn(0x7f94501380a0 0x7f94500a45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:12.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f944ed9d640 1 -- 172.21.15.62:0/759196514 >> v1:172.21.15.62:6789/0 conn(0x7f94500a60b0 legacy=0x7f94500a4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:12.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f944ed9d640 10 monclient: _finish_hunting 0 2024-04-19T01:42:12.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f944ed9d640 1 monclient: found mon.noname-b 2024-04-19T01:42:12.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f944ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:12.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f944ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:42:12.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.918+0000 7f944ed9d640 1 -- 172.21.15.62:0/759196514 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9450073210 con 0x7f9450137780 2024-04-19T01:42:12.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.922+0000 7f944ed9d640 1 -- 172.21.15.62:0/759196514 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9450137780 msgr2=0x7f9450137b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:12.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.922+0000 7f944ed9d640 1 --2- 172.21.15.62:0/759196514 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9450137780 0x7f9450137b60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f94400097c0 tx=0x7f944001c180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51a7e448845a70d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:12.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.922+0000 7f944dd9b640 1 -- 172.21.15.62:0/759196514 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94400b0070 con 0x7f9450137780 2024-04-19T01:42:12.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.922+0000 7f944dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:12.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.922+0000 7f944dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:42:12.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.922+0000 7f944dd9b640 10 monclient: dump: 2024-04-19T01:42:12.929 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:12.929 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:12.929 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:12.929 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:12.929 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:12.929 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:12.929 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:12.929 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:12.929 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:12.929 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:12.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.922+0000 7f944dd9b640 10 monclient: _finish_auth 0 2024-04-19T01:42:12.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.922+0000 7f944dd9b640 10 monclient: _check_auth_tickets 2024-04-19T01:42:12.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.922+0000 7f944dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:12.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.922+0000 7f94551dd640 5 monclient: authenticate success, global_id 4142 2024-04-19T01:42:12.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.922+0000 7f944dd9b640 1 -- 172.21.15.62:0/759196514 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94400ab600 con 0x7f9450137780 2024-04-19T01:42:12.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.922+0000 7f944dd9b640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:12.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.922+0000 7f944dd9b640 1 -- 172.21.15.62:0/759196514 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94400ab9d0 con 0x7f9450137780 2024-04-19T01:42:12.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.922+0000 7f94551dd640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:12.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.922+0000 7f944dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:12.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.922+0000 7f94551dd640 4 set_mon_vals no callback set 2024-04-19T01:42:12.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.922+0000 7f944dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:12.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.922+0000 7f944dd9b640 10 monclient: dump: 2024-04-19T01:42:12.930 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:12.930 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:12.930 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:12.930 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:12.930 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:12.930 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:12.930 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:12.931 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:12.931 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:12.931 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:12.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.922+0000 7f94551dd640 10 monclient: shutdown 2024-04-19T01:42:12.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.922+0000 7f94551dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:12.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.922+0000 7f94551dd640 1 -- 172.21.15.62:0/759196514 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9450137780 msgr2=0x7f9450137b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:12.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.922+0000 7f94551dd640 1 --2- 172.21.15.62:0/759196514 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9450137780 0x7f9450137b60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f94400097c0 tx=0x7f944001c180 comp rx=0 tx=0).stop 2024-04-19T01:42:12.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.922+0000 7f94551dd640 1 -- 172.21.15.62:0/759196514 shutdown_connections 2024-04-19T01:42:12.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.922+0000 7f94551dd640 1 --2- 172.21.15.62:0/759196514 >> v2:172.21.15.62:3300/0 conn(0x7f94501380a0 0x7f94500a45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:12.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.922+0000 7f94551dd640 1 --2- 172.21.15.62:0/759196514 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9450137780 0x7f9450137b60 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:12.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.922+0000 7f94551dd640 1 -- 172.21.15.62:0/759196514 >> 172.21.15.62:0/759196514 conn(0x7f94500aa1d0 msgr2=0x7f94500a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:12.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.922+0000 7f94551dd640 1 -- 172.21.15.62:0/759196514 shutdown_connections 2024-04-19T01:42:12.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.922+0000 7f94551dd640 1 -- 172.21.15.62:0/759196514 wait complete. 2024-04-19T01:42:12.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.922+0000 7f94551dd640 10 monclient: build_initial_monmap 2024-04-19T01:42:12.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.922+0000 7f94551dd640 10 monclient: monmap: 2024-04-19T01:42:12.932 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:12.932 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:12.932 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:12.932 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:12.932 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:12.932 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:12.932 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:12.932 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:12.932 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:12.932 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:12.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.922+0000 7f94551dd640 1 Processor -- start 2024-04-19T01:42:12.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94551dd640 1 -- start start 2024-04-19T01:42:12.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94551dd640 10 monclient: init 2024-04-19T01:42:12.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94551dd640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:12.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94551dd640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-19T01:42:12.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94551dd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f94500a60b0 0x7f945013c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:12.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94551dd640 10 monclient(hunting): picked mon.noname-b con 0x7f94500a60b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:12.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94551dd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9450137780 0x7f9450120220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:12.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94551dd640 10 monclient(hunting): picked mon.noname-c con 0x7f9450137780 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:12.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94551dd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94501380a0 0x7f9450136600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:12.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94551dd640 10 monclient(hunting): picked mon.noname-a con 0x7f94501380a0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:12.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94551dd640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:12.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94551dd640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9450142170 con 0x7f94501380a0 2024-04-19T01:42:12.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94551dd640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:12.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94551dd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f94500a5430 con 0x7f94500a60b0 2024-04-19T01:42:12.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94551dd640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:12.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94551dd640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f94501422f0 con 0x7f9450137780 2024-04-19T01:42:12.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94551dd640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:12.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94551dd640 10 monclient(hunting): authenticate will time out at 918.748663s 2024-04-19T01:42:12.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944e59c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9450137780 0x7f9450120220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:12.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944f59e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94501380a0 0x7f9450136600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:12.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944e59c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9450137780 0x7f9450120220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:46494/0 (socket says 172.21.15.62:46494) 2024-04-19T01:42:12.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944f59e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94501380a0 0x7f9450136600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52802/0 (socket says 172.21.15.62:52802) 2024-04-19T01:42:12.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944e59c640 1 -- 172.21.15.62:0/2178112857 learned_addr learned my addr 172.21.15.62:0/2178112857 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:12.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944e59c640 10 monclient(hunting): get_auth_request con 0x7f9450137780 auth_method 0 2024-04-19T01:42:12.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:12.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944e59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:12.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:12.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944f59e640 10 monclient(hunting): get_auth_request con 0x7f94501380a0 auth_method 0 2024-04-19T01:42:12.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:12.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944f59e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:12.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944f59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:12.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:12.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:12.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:12.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:12.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:12.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:12.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944ed9d640 1 --2- 172.21.15.62:0/2178112857 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f94500a60b0 0x7f945013c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:12.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944ed9d640 10 monclient(hunting): get_auth_request con 0x7f94500a60b0 auth_method 0 2024-04-19T01:42:12.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:12.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:12.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:12.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944e59c640 10 monclient(hunting): handle_auth_done global_id 4145 payload 824 2024-04-19T01:42:12.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944e59c640 1 -- 172.21.15.62:0/2178112857 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f94500a60b0 msgr2=0x7f945013c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:12.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944e59c640 1 --2- 172.21.15.62:0/2178112857 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f94500a60b0 0x7f945013c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:12.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944e59c640 1 -- 172.21.15.62:0/2178112857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94501380a0 msgr2=0x7f9450136600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:12.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944e59c640 1 --2- 172.21.15.62:0/2178112857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94501380a0 0x7f9450136600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:12.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944e59c640 10 monclient: _finish_hunting 0 2024-04-19T01:42:12.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944e59c640 1 monclient: found mon.noname-c 2024-04-19T01:42:12.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:12.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:42:12.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944e59c640 1 -- 172.21.15.62:0/2178112857 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f945013e170 con 0x7f9450137780 2024-04-19T01:42:12.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944e59c640 1 -- 172.21.15.62:0/2178112857 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9450137780 msgr2=0x7f9450120220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:12.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944e59c640 1 --2- 172.21.15.62:0/2178112857 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9450137780 0x7f9450120220 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f944400e930 tx=0x7f944400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:12.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94377fe640 1 -- 172.21.15.62:0/2178112857 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9444017070 con 0x7f9450137780 2024-04-19T01:42:12.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:12.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94377fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:12.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94377fe640 10 monclient: dump: 2024-04-19T01:42:12.938 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:12.938 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:12.938 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:12.938 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:12.938 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:12.939 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:12.939 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:12.939 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:12.939 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:12.939 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:12.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94377fe640 10 monclient: _finish_auth 0 2024-04-19T01:42:12.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94377fe640 10 monclient: _check_auth_tickets 2024-04-19T01:42:12.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:12.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94377fe640 1 -- 172.21.15.62:0/2178112857 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f944400cc10 con 0x7f9450137780 2024-04-19T01:42:12.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94551dd640 5 monclient: authenticate success, global_id 4145 2024-04-19T01:42:12.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94551dd640 10 monclient: _renew_subs 2024-04-19T01:42:12.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94551dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:12.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94551dd640 1 -- 172.21.15.62:0/2178112857 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94500a22f0 con 0x7f9450137780 2024-04-19T01:42:12.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94377fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:12.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94377fe640 1 -- 172.21.15.62:0/2178112857 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9444013470 con 0x7f9450137780 2024-04-19T01:42:12.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:12.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:12.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94377fe640 10 monclient: dump: 2024-04-19T01:42:12.940 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:12.940 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:12.940 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:12.940 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:12.940 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:12.940 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:12.940 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:12.940 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:12.940 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:12.940 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:12.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f944d59a640 4 set_mon_vals no callback set 2024-04-19T01:42:12.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94551dd640 10 monclient: _renew_subs 2024-04-19T01:42:12.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94551dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:12.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94551dd640 1 -- 172.21.15.62:0/2178112857 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f945013ea50 con 0x7f9450137780 2024-04-19T01:42:12.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94377fe640 1 -- 172.21.15.62:0/2178112857 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f944401a070 con 0x7f9450137780 2024-04-19T01:42:12.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.926+0000 7f94377fe640 1 -- 172.21.15.62:0/2178112857 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f9444013cf0 con 0x7f9450137780 2024-04-19T01:42:12.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.930+0000 7f94551dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:12.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.930+0000 7f94551dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:12.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.930+0000 7f94551dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:12.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.930+0000 7f94551dd640 1 -- 172.21.15.62:0/2178112857 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94500692d0 con 0x7f9450137780 2024-04-19T01:42:12.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:12.938+0000 7f944e59c640 1 --2- 172.21.15.62:0/2178112857 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9450137780 0x7f9450120220 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f944400e930 tx=0x7f944400edf0 comp rx=0 tx=0).queue_received will delay after 1.71349e+09 on 0x7f9444013610 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) 2024-04-19T01:42:13.002 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-04-19T01:42:13.005+0000 7f76bc1b9180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-19T01:42:13.171 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-04-19T01:42:13.177+0000 7f76bc1b9180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-19T01:42:13.239 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-04-19T01:42:13.241+0000 7f76bc1b9180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-19T01:42:13.372 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-04-19T01:42:13.377+0000 7f76bc1b9180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-19T01:42:13.397 INFO:tasks.ceph.osd.5.smithi071.stderr:2024-04-19T01:42:13.401+0000 7fdd13a3e7c0 -1 Falling back to public interface 2024-04-19T01:42:13.400 INFO:tasks.ceph.osd.4.smithi071.stderr:2024-04-19T01:42:13.405+0000 7f4b4a89a7c0 -1 Falling back to public interface 2024-04-19T01:42:13.408 INFO:tasks.ceph.osd.3.smithi071.stderr:2024-04-19T01:42:13.413+0000 7f9e34c5a7c0 -1 Falling back to public interface 2024-04-19T01:42:13.420 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-04-19T01:42:13.418+0000 7f797a0797c0 -1 Falling back to public interface 2024-04-19T01:42:13.421 INFO:tasks.ceph.osd.2.smithi062.stderr:2024-04-19T01:42:13.418+0000 7f68ee7587c0 -1 Falling back to public interface 2024-04-19T01:42:13.434 INFO:tasks.ceph.osd.1.smithi062.stderr:2024-04-19T01:42:13.434+0000 7fb13d8cb7c0 -1 Falling back to public interface 2024-04-19T01:42:13.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:13.766+0000 7f94377fe640 1 -- 172.21.15.62:0/2178112857 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f9444013610 con 0x7f9450137780 2024-04-19T01:42:13.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:13.766+0000 7f94377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:13.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:13.766+0000 7f94377fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:13.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:13.830+0000 7f94551dd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-19T01:42:13.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:13.830+0000 7f94551dd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-19T01:42:13.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:13.830+0000 7f94551dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:13.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:13.830+0000 7f94551dd640 1 -- 172.21.15.62:0/2178112857 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9450068ab0 con 0x7f9450137780 2024-04-19T01:42:13.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:13.834+0000 7f94377fe640 1 -- 172.21.15.62:0/2178112857 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7f9444005a40 con 0x7f9450137780 2024-04-19T01:42:13.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:13.834+0000 7f94377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-19T01:42:13.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:13.834+0000 7f94377fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:13.835 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:42:13.835 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":8,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","created":"2024-04-19T01:42:09.115311+0000","modified":"2024-04-19T01:42:12.761455+0000","last_up_change":"0.000000","last_in_change":"2024-04-19T01:42:12.761455+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64304e60-9fe6-413b-8559-3b8ebd179a7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dec72cc0-fa6d-4fe9-b9cd-19dec591ab68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2045906a-e0fd-4c41-b73c-27bf7103243b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"230f6e15-548c-4338-a8ec-067b8f85bd2e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"06789be2-244f-4b6e-ada2-30b6771676af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"552be339-732f-4801-b14d-3d9b08e7f1f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T01:42:13.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:13.834+0000 7f94551dd640 10 monclient: shutdown 2024-04-19T01:42:13.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:13.834+0000 7f94551dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:13.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:13.834+0000 7f94551dd640 1 -- 172.21.15.62:0/2178112857 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9450137780 msgr2=0x7f9450120220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:13.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:13.834+0000 7f94551dd640 1 --2- 172.21.15.62:0/2178112857 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9450137780 0x7f9450120220 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f944400e930 tx=0x7f944400edf0 comp rx=0 tx=0).stop 2024-04-19T01:42:13.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:13.834+0000 7f94551dd640 1 -- 172.21.15.62:0/2178112857 shutdown_connections 2024-04-19T01:42:13.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:13.834+0000 7f94551dd640 1 --2- 172.21.15.62:0/2178112857 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9450137780 0x7f9450120220 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:13.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:13.834+0000 7f94551dd640 1 --2- 172.21.15.62:0/2178112857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94501380a0 0x7f9450136600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:13.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:13.834+0000 7f94551dd640 1 --2- 172.21.15.62:0/2178112857 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f94500a60b0 0x7f945013c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:13.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:13.834+0000 7f94551dd640 1 -- 172.21.15.62:0/2178112857 >> 172.21.15.62:0/2178112857 conn(0x7f94500aa1d0 msgr2=0x7f94500a1b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:13.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:13.834+0000 7f94551dd640 1 -- 172.21.15.62:0/2178112857 shutdown_connections 2024-04-19T01:42:13.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:13.834+0000 7f94551dd640 1 -- 172.21.15.62:0/2178112857 wait complete. 2024-04-19T01:42:13.854 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-19T01:42:13.855 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-19T01:42:13.993 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-04-19T01:42:13.997+0000 7f76bc1b9180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-19T01:42:14.068 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-04-19T01:42:14.073+0000 7f76bc1b9180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-19T01:42:14.247 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-04-19T01:42:14.249+0000 7f76bc1b9180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-19T01:42:14.320 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-04-19T01:42:14.325+0000 7f76bc1b9180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-19T01:42:14.459 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-19T01:42:14.584 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.582+0000 7fa8efc1d640 10 monclient: get_monmap_and_config 2024-04-19T01:42:14.585 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.582+0000 7fa8efc1d640 10 monclient: build_initial_monmap 2024-04-19T01:42:14.585 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.582+0000 7fa8efc1d640 10 monclient: monmap: 2024-04-19T01:42:14.585 INFO:teuthology.misc.health.smithi062.stderr:epoch 0 2024-04-19T01:42:14.585 INFO:teuthology.misc.health.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:14.585 INFO:teuthology.misc.health.smithi062.stderr:last_changed 2024-04-19T01:42:14.588991+0000 2024-04-19T01:42:14.585 INFO:teuthology.misc.health.smithi062.stderr:created 2024-04-19T01:42:14.588991+0000 2024-04-19T01:42:14.585 INFO:teuthology.misc.health.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:14.585 INFO:teuthology.misc.health.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:14.585 INFO:teuthology.misc.health.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:14.585 INFO:teuthology.misc.health.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:14.585 INFO:teuthology.misc.health.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:14.586 INFO:teuthology.misc.health.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:14.586 INFO:teuthology.misc.health.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:14.586 INFO:teuthology.misc.health.smithi062.stderr: 2024-04-19T01:42:14.588 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.582+0000 7fa8efc1d640 1 Processor -- start 2024-04-19T01:42:14.588 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.582+0000 7fa8efc1d640 1 -- start start 2024-04-19T01:42:14.588 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.582+0000 7fa8efc1d640 10 monclient: init 2024-04-19T01:42:14.588 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8efc1d640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:14.588 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8efc1d640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-04-19T01:42:14.588 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8efc1d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa8e8134390 0x7fa8e8134770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:14.588 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8efc1d640 10 monclient(hunting): picked mon.noname-c con 0x7fa8e8134390 addr v2:172.21.15.71:3300/0 2024-04-19T01:42:14.588 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8efc1d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa8e8071ea0 0x7fa8e80a1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:14.588 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8efc1d640 10 monclient(hunting): picked mon.noname-a con 0x7fa8e8071ea0 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:14.589 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8efc1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa8e80a1750 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:14.589 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8efc1d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8e806ece0 con 0x7fa8e80a1750 2024-04-19T01:42:14.589 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8efc1d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:14.589 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8efc1d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fa8e8072f70 con 0x7fa8e8071ea0 2024-04-19T01:42:14.589 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8efc1d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:14.589 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8efc1d640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fa8e8072640 con 0x7fa8e8134390 2024-04-19T01:42:14.589 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8efc1d640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:14.589 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8efc1d640 10 monclient(hunting): authenticate will time out at 920.408429s 2024-04-19T01:42:14.589 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8ed1b8640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa8e8071ea0 0x7fa8e80a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:14.589 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8ee1ba640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fa8e80a1750 0x7fa8e813c380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:45430/0 (socket says 172.21.15.62:45430) 2024-04-19T01:42:14.589 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8ee1ba640 1 -- 172.21.15.62:0/12774848 learned_addr learned my addr 172.21.15.62:0/12774848 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:42:14.589 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8ed1b8640 10 monclient(hunting): get_auth_request con 0x7fa8e8071ea0 auth_method 0 2024-04-19T01:42:14.589 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8ed1b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:14.589 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8ed1b8640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:14.590 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8ed1b8640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:14.590 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8ed1b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:14.590 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8ed1b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:14.590 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8ed1b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:14.590 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8ed9b9640 1 --2- 172.21.15.62:0/12774848 >> v2:172.21.15.71:3300/0 conn(0x7fa8e8134390 0x7fa8e8134770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:14.590 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8ed9b9640 10 monclient(hunting): get_auth_request con 0x7fa8e8134390 auth_method 0 2024-04-19T01:42:14.590 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8ed9b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:14.590 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8ed9b9640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:14.590 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8ed9b9640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:14.590 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8ed1b8640 10 monclient(hunting): handle_auth_done global_id 4131 payload 274 2024-04-19T01:42:14.590 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8ed1b8640 1 -- 172.21.15.62:0/12774848 >> v2:172.21.15.71:3300/0 conn(0x7fa8e8134390 msgr2=0x7fa8e8134770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:14.590 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8ed1b8640 1 --2- 172.21.15.62:0/12774848 >> v2:172.21.15.71:3300/0 conn(0x7fa8e8134390 0x7fa8e8134770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:14.590 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8ed1b8640 1 -- 172.21.15.62:0/12774848 >> v1:172.21.15.62:6789/0 conn(0x7fa8e80a1750 legacy=0x7fa8e813c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:14.590 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8ed1b8640 10 monclient: _finish_hunting 0 2024-04-19T01:42:14.590 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8ed1b8640 1 monclient: found mon.noname-a 2024-04-19T01:42:14.591 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8ed1b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:14.591 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8ed1b8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:14.591 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.586+0000 7fa8ed1b8640 1 -- 172.21.15.62:0/12774848 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8e8072920 con 0x7fa8e8071ea0 2024-04-19T01:42:14.595 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8ed1b8640 1 -- 172.21.15.62:0/12774848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8e8071ea0 msgr2=0x7fa8e80a1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:14.596 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8ed1b8640 1 --2- 172.21.15.62:0/12774848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8e8071ea0 0x7fa8e80a1210 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8073f60 tx=0x7fa8d801c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=337d6a568fabb89e server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:14.596 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8ec9b7640 1 -- 172.21.15.62:0/12774848 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d80b0070 con 0x7fa8e8071ea0 2024-04-19T01:42:14.596 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8ec9b7640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:14.596 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8ec9b7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:14.596 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8ec9b7640 10 monclient: dump: 2024-04-19T01:42:14.596 INFO:teuthology.misc.health.smithi062.stderr:epoch 1 2024-04-19T01:42:14.596 INFO:teuthology.misc.health.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:14.596 INFO:teuthology.misc.health.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:14.596 INFO:teuthology.misc.health.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:14.597 INFO:teuthology.misc.health.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:14.597 INFO:teuthology.misc.health.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:14.597 INFO:teuthology.misc.health.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:14.597 INFO:teuthology.misc.health.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:14.597 INFO:teuthology.misc.health.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:14.597 INFO:teuthology.misc.health.smithi062.stderr: 2024-04-19T01:42:14.597 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8ec9b7640 10 monclient: _finish_auth 0 2024-04-19T01:42:14.597 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8ec9b7640 10 monclient: _check_auth_tickets 2024-04-19T01:42:14.597 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8ec9b7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:14.597 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 5 monclient: authenticate success, global_id 4131 2024-04-19T01:42:14.597 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:42:14.597 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8ec9b7640 1 -- 172.21.15.62:0/12774848 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8d80ab600 con 0x7fa8e8071ea0 2024-04-19T01:42:14.597 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8ec9b7640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:14.597 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:14.597 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8ec9b7640 1 -- 172.21.15.62:0/12774848 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d80ab9d0 con 0x7fa8e8071ea0 2024-04-19T01:42:14.597 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 4 set_mon_vals no callback set 2024-04-19T01:42:14.598 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8ec9b7640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:14.598 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8ec9b7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:14.598 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8ec9b7640 10 monclient: dump: 2024-04-19T01:42:14.598 INFO:teuthology.misc.health.smithi062.stderr:epoch 1 2024-04-19T01:42:14.598 INFO:teuthology.misc.health.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:14.598 INFO:teuthology.misc.health.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:14.598 INFO:teuthology.misc.health.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:14.598 INFO:teuthology.misc.health.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:14.598 INFO:teuthology.misc.health.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:14.598 INFO:teuthology.misc.health.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:14.598 INFO:teuthology.misc.health.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:14.598 INFO:teuthology.misc.health.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:14.598 INFO:teuthology.misc.health.smithi062.stderr: 2024-04-19T01:42:14.599 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 10 monclient: shutdown 2024-04-19T01:42:14.599 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:14.599 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 1 -- 172.21.15.62:0/12774848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8e8071ea0 msgr2=0x7fa8e80a1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:14.599 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 1 --2- 172.21.15.62:0/12774848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8e8071ea0 0x7fa8e80a1210 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8073f60 tx=0x7fa8d801c340 comp rx=0 tx=0).stop 2024-04-19T01:42:14.599 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 1 -- 172.21.15.62:0/12774848 shutdown_connections 2024-04-19T01:42:14.599 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 1 --2- 172.21.15.62:0/12774848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8e8071ea0 0x7fa8e80a1210 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:14.599 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 1 --2- 172.21.15.62:0/12774848 >> v2:172.21.15.71:3300/0 conn(0x7fa8e8134390 0x7fa8e8134770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:14.599 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 1 -- 172.21.15.62:0/12774848 >> 172.21.15.62:0/12774848 conn(0x7fa8e80a2300 msgr2=0x7fa8e80a2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:14.599 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 1 -- 172.21.15.62:0/12774848 shutdown_connections 2024-04-19T01:42:14.599 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 1 -- 172.21.15.62:0/12774848 wait complete. 2024-04-19T01:42:14.599 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 10 monclient: build_initial_monmap 2024-04-19T01:42:14.599 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 10 monclient: monmap: 2024-04-19T01:42:14.599 INFO:teuthology.misc.health.smithi062.stderr:epoch 0 2024-04-19T01:42:14.599 INFO:teuthology.misc.health.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:14.599 INFO:teuthology.misc.health.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:14.600 INFO:teuthology.misc.health.smithi062.stderr:created 0.000000 2024-04-19T01:42:14.600 INFO:teuthology.misc.health.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:14.600 INFO:teuthology.misc.health.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:14.600 INFO:teuthology.misc.health.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:14.600 INFO:teuthology.misc.health.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:14.600 INFO:teuthology.misc.health.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:14.600 INFO:teuthology.misc.health.smithi062.stderr: 2024-04-19T01:42:14.600 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 1 Processor -- start 2024-04-19T01:42:14.600 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 1 -- start start 2024-04-19T01:42:14.600 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 10 monclient: init 2024-04-19T01:42:14.600 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:14.601 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-19T01:42:14.601 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8e8071ea0 0x7fa8e8132270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:14.601 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 10 monclient(hunting): picked mon.noname-b con 0x7fa8e8071ea0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:14.601 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8e80a1750 0x7fa8e81327b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:14.601 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 10 monclient(hunting): picked mon.noname-a con 0x7fa8e80a1750 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:14.601 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa8e8134390 0x7fa8e811be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:14.601 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 10 monclient(hunting): picked mon.noname-c con 0x7fa8e8134390 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:14.601 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:14.601 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e8134e30 con 0x7fa8e80a1750 2024-04-19T01:42:14.601 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:14.601 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e8134cb0 con 0x7fa8e8071ea0 2024-04-19T01:42:14.601 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:14.601 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fa8e80a3080 con 0x7fa8e8134390 2024-04-19T01:42:14.601 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:14.602 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8efc1d640 10 monclient(hunting): authenticate will time out at 920.415819s 2024-04-19T01:42:14.602 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8ed1b8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8e80a1750 0x7fa8e81327b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:14.602 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.590+0000 7fa8ee1ba640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa8e8134390 0x7fa8e811be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:14.602 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ed1b8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8e80a1750 0x7fa8e81327b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52832/0 (socket says 172.21.15.62:52832) 2024-04-19T01:42:14.602 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ed1b8640 1 -- 172.21.15.62:0/2922738056 learned_addr learned my addr 172.21.15.62:0/2922738056 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:14.602 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ed1b8640 10 monclient(hunting): get_auth_request con 0x7fa8e80a1750 auth_method 0 2024-04-19T01:42:14.602 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ed1b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:14.602 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ee1ba640 1 --2- 172.21.15.62:0/2922738056 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa8e8134390 0x7fa8e811be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:46502/0 (socket says 172.21.15.62:46502) 2024-04-19T01:42:14.602 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ed1b8640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:14.602 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ed1b8640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:14.602 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ee1ba640 10 monclient(hunting): get_auth_request con 0x7fa8e8134390 auth_method 0 2024-04-19T01:42:14.602 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ee1ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:14.602 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ee1ba640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:14.602 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ee1ba640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:14.603 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ed1b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:14.603 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ed1b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:14.603 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ed1b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:14.603 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ee1ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:14.603 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ee1ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:14.603 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ed9b9640 1 --2- 172.21.15.62:0/2922738056 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8e8071ea0 0x7fa8e8132270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:14.603 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ee1ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:14.603 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ed9b9640 10 monclient(hunting): get_auth_request con 0x7fa8e8071ea0 auth_method 0 2024-04-19T01:42:14.603 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ed9b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:14.603 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ed9b9640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:14.603 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ed9b9640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:14.603 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ed1b8640 10 monclient(hunting): handle_auth_done global_id 4134 payload 824 2024-04-19T01:42:14.603 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ed1b8640 1 -- 172.21.15.62:0/2922738056 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa8e8134390 msgr2=0x7fa8e811be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:14.603 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ed1b8640 1 --2- 172.21.15.62:0/2922738056 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa8e8134390 0x7fa8e811be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:14.604 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ed1b8640 1 -- 172.21.15.62:0/2922738056 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8e8071ea0 msgr2=0x7fa8e8132270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:14.604 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ed1b8640 1 --2- 172.21.15.62:0/2922738056 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8e8071ea0 0x7fa8e8132270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:14.604 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ed1b8640 10 monclient: _finish_hunting 0 2024-04-19T01:42:14.604 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ed1b8640 1 monclient: found mon.noname-a 2024-04-19T01:42:14.604 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ed1b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:14.604 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ed1b8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:14.604 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ed1b8640 1 -- 172.21.15.62:0/2922738056 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8e811e480 con 0x7fa8e80a1750 2024-04-19T01:42:14.604 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ed9b9640 1 --2- 172.21.15.62:0/2922738056 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8e8071ea0 0x7fa8e8132270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T01:42:14.605 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-04-19T01:42:14.609+0000 7f76bc1b9180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-19T01:42:14.606 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ed1b8640 1 -- 172.21.15.62:0/2922738056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8e80a1750 msgr2=0x7fa8e81327b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:14.606 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8ed1b8640 1 --2- 172.21.15.62:0/2922738056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8e80a1750 0x7fa8e81327b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa8d801a9f0 tx=0x7fa8d80190e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:14.606 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8de7fc640 1 -- 172.21.15.62:0/2922738056 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d80b8020 con 0x7fa8e80a1750 2024-04-19T01:42:14.606 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:14.606 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8de7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:14.606 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8de7fc640 10 monclient: dump: 2024-04-19T01:42:14.607 INFO:teuthology.misc.health.smithi062.stderr:epoch 1 2024-04-19T01:42:14.607 INFO:teuthology.misc.health.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:14.607 INFO:teuthology.misc.health.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:14.607 INFO:teuthology.misc.health.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:14.607 INFO:teuthology.misc.health.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:14.607 INFO:teuthology.misc.health.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:14.607 INFO:teuthology.misc.health.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:14.607 INFO:teuthology.misc.health.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:14.607 INFO:teuthology.misc.health.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:14.607 INFO:teuthology.misc.health.smithi062.stderr: 2024-04-19T01:42:14.607 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8de7fc640 10 monclient: _finish_auth 0 2024-04-19T01:42:14.607 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8de7fc640 10 monclient: _check_auth_tickets 2024-04-19T01:42:14.607 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:14.607 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8de7fc640 1 -- 172.21.15.62:0/2922738056 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8d80b4620 con 0x7fa8e80a1750 2024-04-19T01:42:14.607 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8efc1d640 5 monclient: authenticate success, global_id 4134 2024-04-19T01:42:14.607 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8efc1d640 10 monclient: _renew_subs 2024-04-19T01:42:14.608 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8efc1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:14.608 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8efc1d640 1 -- 172.21.15.62:0/2922738056 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8e80a4710 con 0x7fa8e80a1750 2024-04-19T01:42:14.608 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8de7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:14.608 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8de7fc640 1 -- 172.21.15.62:0/2922738056 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d80b0040 con 0x7fa8e80a1750 2024-04-19T01:42:14.608 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:14.608 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8de7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:14.608 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8de7fc640 10 monclient: dump: 2024-04-19T01:42:14.608 INFO:teuthology.misc.health.smithi062.stderr:epoch 1 2024-04-19T01:42:14.608 INFO:teuthology.misc.health.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:14.608 INFO:teuthology.misc.health.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:14.608 INFO:teuthology.misc.health.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:14.608 INFO:teuthology.misc.health.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:14.608 INFO:teuthology.misc.health.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:14.608 INFO:teuthology.misc.health.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:14.608 INFO:teuthology.misc.health.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:14.609 INFO:teuthology.misc.health.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:14.609 INFO:teuthology.misc.health.smithi062.stderr: 2024-04-19T01:42:14.609 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8dffff640 4 set_mon_vals no callback set 2024-04-19T01:42:14.609 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8efc1d640 10 monclient: _renew_subs 2024-04-19T01:42:14.609 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8efc1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:14.609 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8efc1d640 1 -- 172.21.15.62:0/2922738056 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8e811ed20 con 0x7fa8e80a1750 2024-04-19T01:42:14.609 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8de7fc640 1 -- 172.21.15.62:0/2922738056 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa8d80bb020 con 0x7fa8e80a1750 2024-04-19T01:42:14.609 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8de7fc640 1 -- 172.21.15.62:0/2922738056 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7fa8d80bdae0 con 0x7fa8e80a1750 2024-04-19T01:42:14.609 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8efc1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:14.609 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8efc1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:14.609 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8efc1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:14.609 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.594+0000 7fa8efc1d640 1 -- 172.21.15.62:0/2922738056 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8e80692a0 con 0x7fa8e80a1750 2024-04-19T01:42:14.609 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.598+0000 7fa8ed1b8640 1 --2- 172.21.15.62:0/2922738056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8e80a1750 0x7fa8e81327b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa8d801a9f0 tx=0x7fa8d80190e0 comp rx=0 tx=0).queue_received will delay after 1.71349e+09 on 0x7fa8d80c2030 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) 2024-04-19T01:42:14.665 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-04-19T01:42:14.669+0000 7f76bc1b9180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-19T01:42:14.692 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.690+0000 7fa8de7fc640 1 -- 172.21.15.62:0/2922738056 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fa8d80c2030 con 0x7fa8e80a1750 2024-04-19T01:42:14.692 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.690+0000 7fa8de7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:14.692 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.690+0000 7fa8de7fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:14.762 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.758+0000 7fa8efc1d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-19T01:42:14.762 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.758+0000 7fa8efc1d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-19T01:42:14.762 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.758+0000 7fa8efc1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:14.762 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.758+0000 7fa8efc1d640 1 -- 172.21.15.62:0/2922738056 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa8e806c910 con 0x7fa8e80a1750 2024-04-19T01:42:14.762 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:14.762+0000 7fa8de7fc640 1 -- 172.21.15.62:0/2922738056 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mgrmap(e 2) ==== 44035+0+0 (secure 0 0 0) 0x7fa8d80bd400 con 0x7fa8e80a1750 2024-04-19T01:42:15.426 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:15.426+0000 7fa8de7fc640 1 -- 172.21.15.62:0/2922738056 <== mon.0 v2:172.21.15.62:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7fa8e811e480 con 0x7fa8e80a1750 2024-04-19T01:42:15.426 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:15.426+0000 7fa8de7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-19T01:42:15.427 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:15.426+0000 7fa8de7fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:15.427 INFO:teuthology.misc.health.smithi062.stdout: 2024-04-19T01:42:15.427 INFO:teuthology.misc.health.smithi062.stdout:{"epoch":8,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","created":"2024-04-19T01:42:09.115311+0000","modified":"2024-04-19T01:42:12.761455+0000","last_up_change":"0.000000","last_in_change":"2024-04-19T01:42:12.761455+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"64304e60-9fe6-413b-8559-3b8ebd179a7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dec72cc0-fa6d-4fe9-b9cd-19dec591ab68","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2045906a-e0fd-4c41-b73c-27bf7103243b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"230f6e15-548c-4338-a8ec-067b8f85bd2e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"06789be2-244f-4b6e-ada2-30b6771676af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"552be339-732f-4801-b14d-3d9b08e7f1f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T01:42:15.428 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:15.426+0000 7fa8efc1d640 10 monclient: shutdown 2024-04-19T01:42:15.429 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:15.426+0000 7fa8efc1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:15.429 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:15.426+0000 7fa8efc1d640 1 -- 172.21.15.62:0/2922738056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8e80a1750 msgr2=0x7fa8e81327b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:15.429 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:15.426+0000 7fa8efc1d640 1 --2- 172.21.15.62:0/2922738056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8e80a1750 0x7fa8e81327b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa8d801a9f0 tx=0x7fa8d80190e0 comp rx=0 tx=0).stop 2024-04-19T01:42:15.429 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:15.426+0000 7fa8efc1d640 1 -- 172.21.15.62:0/2922738056 shutdown_connections 2024-04-19T01:42:15.429 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:15.426+0000 7fa8efc1d640 1 --2- 172.21.15.62:0/2922738056 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa8e8134390 0x7fa8e811be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:15.429 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:15.426+0000 7fa8efc1d640 1 --2- 172.21.15.62:0/2922738056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8e80a1750 0x7fa8e81327b0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:15.429 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:15.426+0000 7fa8efc1d640 1 --2- 172.21.15.62:0/2922738056 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8e8071ea0 0x7fa8e8132270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:15.429 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:15.426+0000 7fa8efc1d640 1 -- 172.21.15.62:0/2922738056 >> 172.21.15.62:0/2922738056 conn(0x7fa8e80a2300 msgr2=0x7fa8e80705a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:15.429 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:15.426+0000 7fa8efc1d640 1 -- 172.21.15.62:0/2922738056 shutdown_connections 2024-04-19T01:42:15.429 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:15.426+0000 7fa8efc1d640 1 -- 172.21.15.62:0/2922738056 wait complete. 2024-04-19T01:42:15.454 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-04-19T01:42:15.717 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-04-19T01:42:15.721+0000 7f7663601640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-19T01:42:15.718 INFO:tasks.ceph.mgr.x.smithi071.stderr:2024-04-19T01:42:15.721+0000 7f7663601640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-19T01:42:17.188 INFO:tasks.ceph.osd.5.smithi071.stderr:2024-04-19T01:42:17.193+0000 7fdd13a3e7c0 -1 osd.5 0 log_to_monitors true 2024-04-19T01:42:17.278 INFO:tasks.ceph.osd.3.smithi071.stderr:2024-04-19T01:42:17.281+0000 7f9e34c5a7c0 -1 osd.3 0 log_to_monitors true 2024-04-19T01:42:17.286 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-04-19T01:42:17.286+0000 7f797a0797c0 -1 osd.0 0 log_to_monitors true 2024-04-19T01:42:17.296 INFO:tasks.ceph.osd.4.smithi071.stderr:2024-04-19T01:42:17.301+0000 7f4b4a89a7c0 -1 osd.4 0 log_to_monitors true 2024-04-19T01:42:17.320 INFO:tasks.ceph.osd.1.smithi062.stderr:2024-04-19T01:42:17.318+0000 7fb13d8cb7c0 -1 osd.1 0 log_to_monitors true 2024-04-19T01:42:17.346 INFO:tasks.ceph.osd.2.smithi062.stderr:2024-04-19T01:42:17.346+0000 7f68ee7587c0 -1 osd.2 0 log_to_monitors true 2024-04-19T01:42:19.080 INFO:tasks.ceph.osd.3.smithi071.stderr:2024-04-19T01:42:19.085+0000 7f9e30c03640 -1 osd.3 0 waiting for initial osdmap 2024-04-19T01:42:19.082 INFO:tasks.ceph.osd.1.smithi062.stderr:2024-04-19T01:42:19.082+0000 7fb139874640 -1 osd.1 0 waiting for initial osdmap 2024-04-19T01:42:19.084 INFO:tasks.ceph.osd.2.smithi062.stderr:2024-04-19T01:42:19.082+0000 7f68ea701640 -1 osd.2 0 waiting for initial osdmap 2024-04-19T01:42:19.114 INFO:tasks.ceph.osd.1.smithi062.stderr:2024-04-19T01:42:19.114+0000 7fb1356b0640 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T01:42:19.126 INFO:tasks.ceph.osd.3.smithi071.stderr:2024-04-19T01:42:19.129+0000 7f9e2ca3f640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T01:42:19.132 INFO:tasks.ceph.osd.2.smithi062.stderr:2024-04-19T01:42:19.130+0000 7f68e653d640 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T01:42:19.151 INFO:tasks.ceph.osd.3.smithi071.stderr:2024-04-19T01:42:19.157+0000 7f9e28670640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T01:42:19.895 INFO:tasks.ceph.osd.5.smithi071.stderr:2024-04-19T01:42:19.897+0000 7fdd0f9e7640 -1 osd.5 0 waiting for initial osdmap 2024-04-19T01:42:19.912 INFO:tasks.ceph.osd.5.smithi071.stderr:2024-04-19T01:42:19.917+0000 7fdd0781b640 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T01:42:19.931 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-04-19T01:42:19.930+0000 7f7976022640 -1 osd.0 0 waiting for initial osdmap 2024-04-19T01:42:19.954 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-04-19T01:42:19.954+0000 7f796de56640 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T01:42:20.981 INFO:tasks.ceph.osd.4.smithi071.stderr:2024-04-19T01:42:20.985+0000 7f4b46843640 -1 osd.4 0 waiting for initial osdmap 2024-04-19T01:42:21.013 INFO:tasks.ceph.osd.4.smithi071.stderr:2024-04-19T01:42:21.017+0000 7f4b3e677640 -1 osd.4 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T01:42:22.061 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-19T01:42:22.192 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.190+0000 7fbfe522f640 10 monclient: get_monmap_and_config 2024-04-19T01:42:22.192 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.190+0000 7fbfe522f640 10 monclient: build_initial_monmap 2024-04-19T01:42:22.193 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.190+0000 7fbfe522f640 10 monclient: monmap: 2024-04-19T01:42:22.193 INFO:teuthology.misc.health.smithi062.stderr:epoch 0 2024-04-19T01:42:22.193 INFO:teuthology.misc.health.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:22.193 INFO:teuthology.misc.health.smithi062.stderr:last_changed 2024-04-19T01:42:22.196948+0000 2024-04-19T01:42:22.193 INFO:teuthology.misc.health.smithi062.stderr:created 2024-04-19T01:42:22.196948+0000 2024-04-19T01:42:22.193 INFO:teuthology.misc.health.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:22.193 INFO:teuthology.misc.health.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:22.193 INFO:teuthology.misc.health.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:22.193 INFO:teuthology.misc.health.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:22.193 INFO:teuthology.misc.health.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:22.193 INFO:teuthology.misc.health.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:22.193 INFO:teuthology.misc.health.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:22.193 INFO:teuthology.misc.health.smithi062.stderr: 2024-04-19T01:42:22.196 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.190+0000 7fbfe522f640 1 Processor -- start 2024-04-19T01:42:22.196 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfe522f640 1 -- start start 2024-04-19T01:42:22.196 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfe522f640 10 monclient: init 2024-04-19T01:42:22.196 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfe522f640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:22.196 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfe522f640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-04-19T01:42:22.196 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfe522f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbfe01378e0 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:22.196 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfe522f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbfe009c8b0 0x7fbfe013c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:22.196 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfe522f640 10 monclient(hunting): picked mon.noname-a con 0x7fbfe009c8b0 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:22.196 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfe522f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbfe013cb50 0x7fbfe013ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:22.196 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfe522f640 10 monclient(hunting): picked mon.noname-b con 0x7fbfe013cb50 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:22.196 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfe522f640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfe00a62b0 con 0x7fbfe01378e0 2024-04-19T01:42:22.196 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfe522f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:22.196 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfe522f640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fbfe00a6000 con 0x7fbfe009c8b0 2024-04-19T01:42:22.197 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfe522f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:22.197 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfe522f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fbfe009bbf0 con 0x7fbfe013cb50 2024-04-19T01:42:22.197 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfe522f640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:22.197 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfe522f640 10 monclient(hunting): authenticate will time out at 928.016846s 2024-04-19T01:42:22.197 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfde59c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbfe009c8b0 0x7fbfe013c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:22.197 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfdf59e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbfe013cb50 0x7fbfe013ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:22.197 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfded9d640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fbfe01378e0 0x7fbfe0137cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:60974/0 (socket says 172.21.15.62:60974) 2024-04-19T01:42:22.197 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfde59c640 10 monclient(hunting): get_auth_request con 0x7fbfe009c8b0 auth_method 0 2024-04-19T01:42:22.197 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfded9d640 1 -- 172.21.15.62:0/3614619006 learned_addr learned my addr 172.21.15.62:0/3614619006 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:42:22.197 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfde59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:22.197 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfde59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:22.197 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfde59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:22.197 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfdf59e640 10 monclient(hunting): get_auth_request con 0x7fbfe013cb50 auth_method 0 2024-04-19T01:42:22.197 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfdf59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:22.198 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfdf59e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:22.198 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfdf59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:22.198 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfde59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:22.198 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfde59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:22.198 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfde59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:22.198 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfdf59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:22.198 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfdf59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:22.198 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfdf59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:22.198 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfde59c640 10 monclient(hunting): handle_auth_done global_id 4170 payload 274 2024-04-19T01:42:22.198 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfde59c640 1 -- 172.21.15.62:0/3614619006 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbfe013cb50 msgr2=0x7fbfe013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:22.198 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfde59c640 1 --2- 172.21.15.62:0/3614619006 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbfe013cb50 0x7fbfe013ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:22.198 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfde59c640 1 -- 172.21.15.62:0/3614619006 >> v1:172.21.15.62:6789/0 conn(0x7fbfe01378e0 legacy=0x7fbfe0137cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:22.198 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfde59c640 10 monclient: _finish_hunting 0 2024-04-19T01:42:22.198 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfde59c640 1 monclient: found mon.noname-a 2024-04-19T01:42:22.198 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfde59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:22.199 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfde59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:22.199 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfde59c640 1 -- 172.21.15.62:0/3614619006 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfe009bed0 con 0x7fbfe009c8b0 2024-04-19T01:42:22.199 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.194+0000 7fbfdf59e640 1 --2- 172.21.15.62:0/3614619006 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbfe013cb50 0x7fbfe013ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:42:22.200 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfde59c640 1 -- 172.21.15.62:0/3614619006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfe009c8b0 msgr2=0x7fbfe013c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:22.201 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfde59c640 1 --2- 172.21.15.62:0/3614619006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfe009c8b0 0x7fbfe013c610 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbfd40097c0 tx=0x7fbfd401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5410341c975bc79 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:22.201 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfddd9b640 1 -- 172.21.15.62:0/3614619006 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd40b0070 con 0x7fbfe009c8b0 2024-04-19T01:42:22.201 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfddd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:22.201 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfddd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:22.201 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfddd9b640 10 monclient: dump: 2024-04-19T01:42:22.201 INFO:teuthology.misc.health.smithi062.stderr:epoch 1 2024-04-19T01:42:22.201 INFO:teuthology.misc.health.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:22.201 INFO:teuthology.misc.health.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:22.201 INFO:teuthology.misc.health.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:22.201 INFO:teuthology.misc.health.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:22.201 INFO:teuthology.misc.health.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:22.201 INFO:teuthology.misc.health.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:22.201 INFO:teuthology.misc.health.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:22.202 INFO:teuthology.misc.health.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:22.202 INFO:teuthology.misc.health.smithi062.stderr: 2024-04-19T01:42:22.202 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfddd9b640 10 monclient: _finish_auth 0 2024-04-19T01:42:22.202 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfddd9b640 10 monclient: _check_auth_tickets 2024-04-19T01:42:22.202 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfddd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:22.202 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfddd9b640 1 -- 172.21.15.62:0/3614619006 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfd40ab400 con 0x7fbfe009c8b0 2024-04-19T01:42:22.202 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfddd9b640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:22.202 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfddd9b640 1 -- 172.21.15.62:0/3614619006 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd40ab7d0 con 0x7fbfe009c8b0 2024-04-19T01:42:22.202 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfe522f640 5 monclient: authenticate success, global_id 4170 2024-04-19T01:42:22.202 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfe522f640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:22.202 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfe522f640 4 set_mon_vals no callback set 2024-04-19T01:42:22.202 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfddd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:22.202 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfddd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:22.202 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfddd9b640 10 monclient: dump: 2024-04-19T01:42:22.203 INFO:teuthology.misc.health.smithi062.stderr:epoch 1 2024-04-19T01:42:22.203 INFO:teuthology.misc.health.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:22.203 INFO:teuthology.misc.health.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:22.203 INFO:teuthology.misc.health.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:22.203 INFO:teuthology.misc.health.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:22.203 INFO:teuthology.misc.health.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:22.203 INFO:teuthology.misc.health.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:22.203 INFO:teuthology.misc.health.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:22.203 INFO:teuthology.misc.health.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:22.203 INFO:teuthology.misc.health.smithi062.stderr: 2024-04-19T01:42:22.203 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfe522f640 10 monclient: shutdown 2024-04-19T01:42:22.203 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfe522f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:22.203 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfe522f640 1 -- 172.21.15.62:0/3614619006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfe009c8b0 msgr2=0x7fbfe013c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:22.203 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfe522f640 1 --2- 172.21.15.62:0/3614619006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfe009c8b0 0x7fbfe013c610 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbfd40097c0 tx=0x7fbfd401c3f0 comp rx=0 tx=0).stop 2024-04-19T01:42:22.203 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfe522f640 1 -- 172.21.15.62:0/3614619006 shutdown_connections 2024-04-19T01:42:22.204 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfe522f640 1 --2- 172.21.15.62:0/3614619006 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbfe013cb50 0x7fbfe013ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:22.204 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfe522f640 1 --2- 172.21.15.62:0/3614619006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfe009c8b0 0x7fbfe013c610 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:22.204 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfe522f640 1 -- 172.21.15.62:0/3614619006 >> 172.21.15.62:0/3614619006 conn(0x7fbfe0073c70 msgr2=0x7fbfe00a9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:22.204 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfe522f640 1 -- 172.21.15.62:0/3614619006 shutdown_connections 2024-04-19T01:42:22.204 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfe522f640 1 -- 172.21.15.62:0/3614619006 wait complete. 2024-04-19T01:42:22.204 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfe522f640 10 monclient: build_initial_monmap 2024-04-19T01:42:22.204 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfe522f640 10 monclient: monmap: 2024-04-19T01:42:22.204 INFO:teuthology.misc.health.smithi062.stderr:epoch 0 2024-04-19T01:42:22.204 INFO:teuthology.misc.health.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:22.204 INFO:teuthology.misc.health.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:22.204 INFO:teuthology.misc.health.smithi062.stderr:created 0.000000 2024-04-19T01:42:22.204 INFO:teuthology.misc.health.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:22.204 INFO:teuthology.misc.health.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:22.204 INFO:teuthology.misc.health.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:22.205 INFO:teuthology.misc.health.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:22.205 INFO:teuthology.misc.health.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:22.205 INFO:teuthology.misc.health.smithi062.stderr: 2024-04-19T01:42:22.205 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfe522f640 1 Processor -- start 2024-04-19T01:42:22.205 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfe522f640 1 -- start start 2024-04-19T01:42:22.205 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.198+0000 7fbfe522f640 10 monclient: init 2024-04-19T01:42:22.209 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfe522f640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:22.209 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfe522f640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-19T01:42:22.209 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfe522f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfe009c8b0 0x7fbfe01cf410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:22.209 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfe522f640 10 monclient(hunting): picked mon.noname-a con 0x7fbfe009c8b0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:22.209 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfe522f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbfe01378e0 0x7fbfe00a2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:22.209 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfe522f640 10 monclient(hunting): picked mon.noname-c con 0x7fbfe01378e0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:22.209 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfe522f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfe013cb50 0x7fbfe00a2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:22.209 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfe522f640 10 monclient(hunting): picked mon.noname-b con 0x7fbfe013cb50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:22.209 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfe522f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:22.209 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfe522f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe0141eb0 con 0x7fbfe009c8b0 2024-04-19T01:42:22.209 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfe522f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:22.210 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfe522f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe009b770 con 0x7fbfe013cb50 2024-04-19T01:42:22.210 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfe522f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:22.210 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfe522f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fbfe0142030 con 0x7fbfe01378e0 2024-04-19T01:42:22.210 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfe522f640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:22.210 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfe522f640 10 monclient(hunting): authenticate will time out at 928.024336s 2024-04-19T01:42:22.210 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfded9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfe009c8b0 0x7fbfe01cf410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:22.210 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfde59c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbfe01378e0 0x7fbfe00a2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:22.210 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfded9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfe009c8b0 0x7fbfe01cf410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52388/0 (socket says 172.21.15.62:52388) 2024-04-19T01:42:22.210 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfde59c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbfe01378e0 0x7fbfe00a2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:57756/0 (socket says 172.21.15.62:57756) 2024-04-19T01:42:22.210 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfded9d640 1 -- 172.21.15.62:0/2854568836 learned_addr learned my addr 172.21.15.62:0/2854568836 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:22.210 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfded9d640 10 monclient(hunting): get_auth_request con 0x7fbfe009c8b0 auth_method 0 2024-04-19T01:42:22.210 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfded9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:22.210 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfded9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:22.210 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfded9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:22.210 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfde59c640 10 monclient(hunting): get_auth_request con 0x7fbfe01378e0 auth_method 0 2024-04-19T01:42:22.211 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfde59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:22.211 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfde59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:22.211 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfde59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:22.211 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfde59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:22.211 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfde59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:22.211 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfde59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:22.211 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfded9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:22.211 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfded9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:22.211 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfded9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:22.211 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfdf59e640 1 --2- 172.21.15.62:0/2854568836 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfe013cb50 0x7fbfe00a2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:22.211 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfdf59e640 10 monclient(hunting): get_auth_request con 0x7fbfe013cb50 auth_method 0 2024-04-19T01:42:22.211 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfdf59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:22.211 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfdf59e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:22.211 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfdf59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:22.212 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfde59c640 10 monclient(hunting): handle_auth_done global_id 4184 payload 824 2024-04-19T01:42:22.212 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfde59c640 1 -- 172.21.15.62:0/2854568836 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfe013cb50 msgr2=0x7fbfe00a2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:22.212 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfde59c640 1 --2- 172.21.15.62:0/2854568836 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfe013cb50 0x7fbfe00a2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:22.212 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfde59c640 1 -- 172.21.15.62:0/2854568836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfe009c8b0 msgr2=0x7fbfe01cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:22.212 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfde59c640 1 --2- 172.21.15.62:0/2854568836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfe009c8b0 0x7fbfe01cf410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:22.212 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfde59c640 10 monclient: _finish_hunting 0 2024-04-19T01:42:22.212 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfde59c640 1 monclient: found mon.noname-c 2024-04-19T01:42:22.212 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfde59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:22.212 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfde59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:42:22.212 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfde59c640 1 -- 172.21.15.62:0/2854568836 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfe009f0d0 con 0x7fbfe01378e0 2024-04-19T01:42:22.212 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfde59c640 1 -- 172.21.15.62:0/2854568836 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbfe01378e0 msgr2=0x7fbfe00a2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:22.212 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfde59c640 1 --2- 172.21.15.62:0/2854568836 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbfe01378e0 0x7fbfe00a2240 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbfd40ab000 tx=0x7fbfd4019190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:22.212 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfc77fe640 1 -- 172.21.15.62:0/2854568836 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd40b8030 con 0x7fbfe01378e0 2024-04-19T01:42:22.212 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfc77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:22.212 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfc77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:22.213 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfc77fe640 10 monclient: dump: 2024-04-19T01:42:22.213 INFO:teuthology.misc.health.smithi062.stderr:epoch 1 2024-04-19T01:42:22.213 INFO:teuthology.misc.health.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:22.213 INFO:teuthology.misc.health.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:22.213 INFO:teuthology.misc.health.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:22.213 INFO:teuthology.misc.health.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:22.213 INFO:teuthology.misc.health.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:22.213 INFO:teuthology.misc.health.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:22.213 INFO:teuthology.misc.health.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:22.213 INFO:teuthology.misc.health.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:22.213 INFO:teuthology.misc.health.smithi062.stderr: 2024-04-19T01:42:22.213 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfc77fe640 10 monclient: _finish_auth 0 2024-04-19T01:42:22.213 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfc77fe640 10 monclient: _check_auth_tickets 2024-04-19T01:42:22.213 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfc77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:22.213 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfc77fe640 1 -- 172.21.15.62:0/2854568836 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfd40b4410 con 0x7fbfe01378e0 2024-04-19T01:42:22.213 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfc77fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:22.214 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfc77fe640 1 -- 172.21.15.62:0/2854568836 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd40b0040 con 0x7fbfe01378e0 2024-04-19T01:42:22.214 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfc77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:22.214 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfc77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:22.214 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfc77fe640 10 monclient: dump: 2024-04-19T01:42:22.214 INFO:teuthology.misc.health.smithi062.stderr:epoch 1 2024-04-19T01:42:22.214 INFO:teuthology.misc.health.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:22.214 INFO:teuthology.misc.health.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:22.214 INFO:teuthology.misc.health.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:22.214 INFO:teuthology.misc.health.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:22.214 INFO:teuthology.misc.health.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:22.214 INFO:teuthology.misc.health.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:22.214 INFO:teuthology.misc.health.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:22.214 INFO:teuthology.misc.health.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:22.214 INFO:teuthology.misc.health.smithi062.stderr: 2024-04-19T01:42:22.215 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfe522f640 5 monclient: authenticate success, global_id 4184 2024-04-19T01:42:22.215 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfe522f640 10 monclient: _renew_subs 2024-04-19T01:42:22.215 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfe522f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:22.215 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfe522f640 1 -- 172.21.15.62:0/2854568836 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfe00a3360 con 0x7fbfe01378e0 2024-04-19T01:42:22.215 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfdd59a640 4 set_mon_vals no callback set 2024-04-19T01:42:22.215 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfe522f640 10 monclient: _renew_subs 2024-04-19T01:42:22.215 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfe522f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:22.215 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.202+0000 7fbfe522f640 1 -- 172.21.15.62:0/2854568836 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfe01cfb30 con 0x7fbfe01378e0 2024-04-19T01:42:22.215 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.206+0000 7fbfc77fe640 1 -- 172.21.15.62:0/2854568836 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 4) ==== 44268+0+0 (secure 0 0 0) 0x7fbfd40b4880 con 0x7fbfe01378e0 2024-04-19T01:42:22.215 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.206+0000 7fbfe522f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:22.215 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.206+0000 7fbfe522f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:22.215 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.206+0000 7fbfe522f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:22.215 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.206+0000 7fbfe522f640 1 -- 172.21.15.62:0/2854568836 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfe00692a0 con 0x7fbfe01378e0 2024-04-19T01:42:22.215 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.206+0000 7fbfc77fe640 1 --2- 172.21.15.62:0/2854568836 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fbfcc0363e0 0x7fbfcc0388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:22.215 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.206+0000 7fbfc77fe640 1 -- 172.21.15.62:0/2854568836 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(13..13 src has 1..13) ==== 3993+0+0 (secure 0 0 0) 0x7fbfd40e26a0 con 0x7fbfe01378e0 2024-04-19T01:42:22.215 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.206+0000 7fbfded9d640 1 --2- 172.21.15.62:0/2854568836 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fbfcc0363e0 0x7fbfcc0388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:22.216 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.206+0000 7fbfded9d640 10 monclient: get_auth_request con 0x7fbfcc0363e0 auth_method 0 2024-04-19T01:42:22.216 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.206+0000 7fbfded9d640 1 --2- 172.21.15.62:0/2854568836 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fbfcc0363e0 0x7fbfcc0388a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0007d60 tx=0x7fbfd0007cb0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:22.216 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.210+0000 7fbfc77fe640 1 -- 172.21.15.62:0/2854568836 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fbfd4016070 con 0x7fbfe01378e0 2024-04-19T01:42:22.216 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.210+0000 7fbfc77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:22.216 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.210+0000 7fbfc77fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:22.388 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.386+0000 7fbfe522f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-19T01:42:22.388 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.386+0000 7fbfe522f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-19T01:42:22.388 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.386+0000 7fbfe522f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:22.388 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.386+0000 7fbfe522f640 1 -- 172.21.15.62:0/2854568836 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbfe00685f0 con 0x7fbfe01378e0 2024-04-19T01:42:22.389 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.386+0000 7fbfc77fe640 1 -- 172.21.15.62:0/2854568836 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) ==== 74+0+10438 (secure 0 0 0) 0x7fbfd40e13c0 con 0x7fbfe01378e0 2024-04-19T01:42:22.389 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.386+0000 7fbfc77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-19T01:42:22.389 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.386+0000 7fbfc77fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:22.389 INFO:teuthology.misc.health.smithi062.stdout: 2024-04-19T01:42:22.389 INFO:teuthology.misc.health.smithi062.stdout:{"epoch":13,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","created":"2024-04-19T01:42:09.115311+0000","modified":"2024-04-19T01:42:22.112729+0000","last_up_change":"2024-04-19T01:42:21.102542+0000","last_in_change":"2024-04-19T01:42:12.761455+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-19T01:42:20.748429+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"64304e60-9fe6-413b-8559-3b8ebd179a7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":3870805292},{"type":"v1","addr":"172.21.15.62:6802","nonce":3870805292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":3870805292},{"type":"v1","addr":"172.21.15.62:6806","nonce":3870805292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":3870805292},{"type":"v1","addr":"172.21.15.62:6814","nonce":3870805292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":3870805292},{"type":"v1","addr":"172.21.15.62:6810","nonce":3870805292}]},"public_addr":"172.21.15.62:6802/3870805292","cluster_addr":"172.21.15.62:6806/3870805292","heartbeat_back_addr":"172.21.15.62:6814/3870805292","heartbeat_front_addr":"172.21.15.62:6810/3870805292","state":["exists","up"]},{"osd":1,"uuid":"dec72cc0-fa6d-4fe9-b9cd-19dec591ab68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":199109455},{"type":"v1","addr":"172.21.15.62:6817","nonce":199109455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":199109455},{"type":"v1","addr":"172.21.15.62:6819","nonce":199109455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":199109455},{"type":"v1","addr":"172.21.15.62:6823","nonce":199109455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":199109455},{"type":"v1","addr":"172.21.15.62:6821","nonce":199109455}]},"public_addr":"172.21.15.62:6817/199109455","cluster_addr":"172.21.15.62:6819/199109455","heartbeat_back_addr":"172.21.15.62:6823/199109455","heartbeat_front_addr":"172.21.15.62:6821/199109455","state":["exists","up"]},{"osd":2,"uuid":"2045906a-e0fd-4c41-b73c-27bf7103243b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6801","nonce":1128256055},{"type":"v1","addr":"172.21.15.62:6803","nonce":1128256055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6805","nonce":1128256055},{"type":"v1","addr":"172.21.15.62:6807","nonce":1128256055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6813","nonce":1128256055},{"type":"v1","addr":"172.21.15.62:6815","nonce":1128256055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6809","nonce":1128256055},{"type":"v1","addr":"172.21.15.62:6811","nonce":1128256055}]},"public_addr":"172.21.15.62:6803/1128256055","cluster_addr":"172.21.15.62:6807/1128256055","heartbeat_back_addr":"172.21.15.62:6815/1128256055","heartbeat_front_addr":"172.21.15.62:6811/1128256055","state":["exists","up"]},{"osd":3,"uuid":"230f6e15-548c-4338-a8ec-067b8f85bd2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":2632024688},{"type":"v1","addr":"172.21.15.71:6817","nonce":2632024688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":2632024688},{"type":"v1","addr":"172.21.15.71:6819","nonce":2632024688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":2632024688},{"type":"v1","addr":"172.21.15.71:6823","nonce":2632024688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":2632024688},{"type":"v1","addr":"172.21.15.71:6821","nonce":2632024688}]},"public_addr":"172.21.15.71:6817/2632024688","cluster_addr":"172.21.15.71:6819/2632024688","heartbeat_back_addr":"172.21.15.71:6823/2632024688","heartbeat_front_addr":"172.21.15.71:6821/2632024688","state":["exists","up"]},{"osd":4,"uuid":"06789be2-244f-4b6e-ada2-30b6771676af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":3342848064},{"type":"v1","addr":"172.21.15.71:6809","nonce":3342848064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3342848064},{"type":"v1","addr":"172.21.15.71:6811","nonce":3342848064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":3342848064},{"type":"v1","addr":"172.21.15.71:6815","nonce":3342848064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":3342848064},{"type":"v1","addr":"172.21.15.71:6813","nonce":3342848064}]},"public_addr":"172.21.15.71:6809/3342848064","cluster_addr":"172.21.15.71:6811/3342848064","heartbeat_back_addr":"172.21.15.71:6815/3342848064","heartbeat_front_addr":"172.21.15.71:6813/3342848064","state":["exists","up"]},{"osd":5,"uuid":"552be339-732f-4801-b14d-3d9b08e7f1f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":196330660},{"type":"v1","addr":"172.21.15.71:6801","nonce":196330660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":196330660},{"type":"v1","addr":"172.21.15.71:6803","nonce":196330660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":196330660},{"type":"v1","addr":"172.21.15.71:6807","nonce":196330660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":196330660},{"type":"v1","addr":"172.21.15.71:6805","nonce":196330660}]},"public_addr":"172.21.15.71:6801/196330660","cluster_addr":"172.21.15.71:6803/196330660","heartbeat_back_addr":"172.21.15.71:6807/196330660","heartbeat_front_addr":"172.21.15.71:6805/196330660","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:42:18.255067+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:42:18.295998+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:42:18.304269+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:42:18.274604+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:42:18.338084+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:42:18.149308+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T01:42:22.391 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.390+0000 7fbfe522f640 1 -- 172.21.15.62:0/2854568836 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fbfcc0363e0 msgr2=0x7fbfcc0388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:22.391 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.390+0000 7fbfe522f640 1 --2- 172.21.15.62:0/2854568836 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fbfcc0363e0 0x7fbfcc0388a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0007d60 tx=0x7fbfd0007cb0 comp rx=0 tx=0).stop 2024-04-19T01:42:22.391 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.390+0000 7fbfe522f640 10 monclient: shutdown 2024-04-19T01:42:22.392 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.390+0000 7fbfe522f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:22.392 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.390+0000 7fbfe522f640 1 -- 172.21.15.62:0/2854568836 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbfe01378e0 msgr2=0x7fbfe00a2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:22.392 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.390+0000 7fbfe522f640 1 --2- 172.21.15.62:0/2854568836 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbfe01378e0 0x7fbfe00a2240 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbfd40ab000 tx=0x7fbfd4019190 comp rx=0 tx=0).stop 2024-04-19T01:42:22.392 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.390+0000 7fbfe522f640 1 -- 172.21.15.62:0/2854568836 shutdown_connections 2024-04-19T01:42:22.392 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.390+0000 7fbfe522f640 1 --2- 172.21.15.62:0/2854568836 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fbfcc0363e0 0x7fbfcc0388a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:22.392 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.390+0000 7fbfe522f640 1 --2- 172.21.15.62:0/2854568836 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbfe01378e0 0x7fbfe00a2240 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:22.392 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.390+0000 7fbfe522f640 1 --2- 172.21.15.62:0/2854568836 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfe013cb50 0x7fbfe00a2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:22.392 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.390+0000 7fbfe522f640 1 --2- 172.21.15.62:0/2854568836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfe009c8b0 0x7fbfe01cf410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:22.392 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.390+0000 7fbfe522f640 1 -- 172.21.15.62:0/2854568836 >> 172.21.15.62:0/2854568836 conn(0x7fbfe0073c70 msgr2=0x7fbfe013d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:22.392 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.390+0000 7fbfe522f640 1 -- 172.21.15.62:0/2854568836 shutdown_connections 2024-04-19T01:42:22.392 INFO:teuthology.misc.health.smithi062.stderr:2024-04-19T01:42:22.390+0000 7fbfe522f640 1 -- 172.21.15.62:0/2854568836 wait complete. 2024-04-19T01:42:22.409 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-04-19T01:42:22.409 INFO:tasks.ceph:Creating RBD pool 2024-04-19T01:42:22.409 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-19T01:42:22.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d3e5bc640 10 monclient: get_monmap_and_config 2024-04-19T01:42:22.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d3e5bc640 10 monclient: build_initial_monmap 2024-04-19T01:42:22.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d3e5bc640 10 monclient: monmap: 2024-04-19T01:42:22.526 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:22.526 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:22.526 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:22.530175+0000 2024-04-19T01:42:22.526 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:22.530175+0000 2024-04-19T01:42:22.526 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:22.526 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:22.526 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:22.527 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:22.527 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:22.527 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:22.527 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:22.527 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:22.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d3e5bc640 1 Processor -- start 2024-04-19T01:42:22.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d3e5bc640 1 -- start start 2024-04-19T01:42:22.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d3e5bc640 10 monclient: init 2024-04-19T01:42:22.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d3e5bc640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:22.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d3e5bc640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-04-19T01:42:22.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d3e5bc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4d380a3c60 0x7f4d380a2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:22.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d3e5bc640 10 monclient(hunting): picked mon.noname-a con 0x7f4d380a3c60 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:22.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d3e5bc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4d380a29a0 addr v1:172.21.15.71:6789/0 2024-04-19T01:42:22.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d3e5bc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4d380a2dd0 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:22.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d3e5bc640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d38135030 con 0x7f4d380a2dd0 2024-04-19T01:42:22.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d3e5bc640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d38076170 con 0x7f4d380a29a0 2024-04-19T01:42:22.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d3e5bc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:22.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d3e5bc640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f4d3809ff80 con 0x7f4d380a3c60 2024-04-19T01:42:22.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d3e5bc640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:22.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d3e5bc640 10 monclient(hunting): authenticate will time out at 928.349523s 2024-04-19T01:42:22.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d37fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4d380a3c60 0x7f4d380a2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:22.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d37fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4d380a3c60 0x7f4d380a2460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52398/0 (socket says 172.21.15.62:52398) 2024-04-19T01:42:22.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d37fff640 1 -- 172.21.15.62:0/4108269889 learned_addr learned my addr 172.21.15.62:0/4108269889 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:22.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d3cb59640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f4d380a2dd0 0x7f4d38134920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:60984/0 (socket says 172.21.15.62:60984) 2024-04-19T01:42:22.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d37fff640 10 monclient(hunting): get_auth_request con 0x7f4d380a3c60 auth_method 0 2024-04-19T01:42:22.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:22.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d37fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:22.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d37fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:22.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:22.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:22.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:22.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d37fff640 10 monclient(hunting): handle_auth_done global_id 4182 payload 274 2024-04-19T01:42:22.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d37fff640 1 -- 172.21.15.62:0/4108269889 >> v1:172.21.15.71:6789/0 conn(0x7f4d380a29a0 legacy=0x7f4d38134210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:22.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d37fff640 1 -- 172.21.15.62:0/4108269889 >> v1:172.21.15.62:6789/0 conn(0x7f4d380a2dd0 legacy=0x7f4d38134920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:22.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d37fff640 10 monclient: _finish_hunting 0 2024-04-19T01:42:22.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d37fff640 1 monclient: found mon.noname-a 2024-04-19T01:42:22.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:22.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d37fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:22.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.526+0000 7f4d37fff640 1 -- 172.21.15.62:0/4108269889 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d380a0800 con 0x7f4d380a3c60 2024-04-19T01:42:22.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.530+0000 7f4d37fff640 1 -- 172.21.15.62:0/4108269889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d380a3c60 msgr2=0x7f4d380a2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:22.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.530+0000 7f4d37fff640 1 --2- 172.21.15.62:0/4108269889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d380a3c60 0x7f4d380a2460 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c009870 tx=0x7f4d2c01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18c36ea7aa1e4f9e server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:22.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.530+0000 7f4d36ffd640 1 -- 172.21.15.62:0/4108269889 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c0b0070 con 0x7f4d380a3c60 2024-04-19T01:42:22.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.530+0000 7f4d36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:22.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.530+0000 7f4d36ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:22.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.530+0000 7f4d36ffd640 10 monclient: dump: 2024-04-19T01:42:22.536 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:22.536 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:22.537 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:22.537 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:22.537 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:22.537 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:22.537 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:22.537 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:22.537 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:22.537 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:22.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.530+0000 7f4d36ffd640 10 monclient: _finish_auth 0 2024-04-19T01:42:22.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.530+0000 7f4d36ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:42:22.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.530+0000 7f4d36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:22.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.530+0000 7f4d36ffd640 1 -- 172.21.15.62:0/4108269889 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d2c0ab600 con 0x7f4d380a3c60 2024-04-19T01:42:22.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.530+0000 7f4d3e5bc640 5 monclient: authenticate success, global_id 4182 2024-04-19T01:42:22.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.530+0000 7f4d3e5bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:42:22.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.530+0000 7f4d36ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:22.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.530+0000 7f4d3e5bc640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:22.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.530+0000 7f4d36ffd640 1 -- 172.21.15.62:0/4108269889 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c0ab9d0 con 0x7f4d380a3c60 2024-04-19T01:42:22.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.530+0000 7f4d36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:22.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.530+0000 7f4d3e5bc640 4 set_mon_vals no callback set 2024-04-19T01:42:22.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.530+0000 7f4d36ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:22.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.530+0000 7f4d36ffd640 10 monclient: dump: 2024-04-19T01:42:22.538 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:22.538 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:22.539 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:22.539 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:22.539 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:22.539 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:22.539 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:22.539 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:22.539 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:22.539 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:22.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.530+0000 7f4d3e5bc640 10 monclient: shutdown 2024-04-19T01:42:22.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.530+0000 7f4d3e5bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:22.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.530+0000 7f4d3e5bc640 1 -- 172.21.15.62:0/4108269889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d380a3c60 msgr2=0x7f4d380a2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:22.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.530+0000 7f4d3e5bc640 1 --2- 172.21.15.62:0/4108269889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d380a3c60 0x7f4d380a2460 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c009870 tx=0x7f4d2c01c420 comp rx=0 tx=0).stop 2024-04-19T01:42:22.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.530+0000 7f4d3e5bc640 1 -- 172.21.15.62:0/4108269889 shutdown_connections 2024-04-19T01:42:22.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.530+0000 7f4d3e5bc640 1 --2- 172.21.15.62:0/4108269889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d380a3c60 0x7f4d380a2460 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:22.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.530+0000 7f4d3e5bc640 1 -- 172.21.15.62:0/4108269889 >> 172.21.15.62:0/4108269889 conn(0x7f4d380a7050 msgr2=0x7f4d380a3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:22.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.530+0000 7f4d3e5bc640 1 -- 172.21.15.62:0/4108269889 shutdown_connections 2024-04-19T01:42:22.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.530+0000 7f4d3e5bc640 1 -- 172.21.15.62:0/4108269889 wait complete. 2024-04-19T01:42:22.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.530+0000 7f4d3e5bc640 10 monclient: build_initial_monmap 2024-04-19T01:42:22.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.530+0000 7f4d3e5bc640 10 monclient: monmap: 2024-04-19T01:42:22.540 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:22.540 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:22.540 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:22.540 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:22.540 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:22.540 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:22.541 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:22.541 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:22.541 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:22.541 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:22.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d3e5bc640 1 Processor -- start 2024-04-19T01:42:22.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d3e5bc640 1 -- start start 2024-04-19T01:42:22.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d3e5bc640 10 monclient: init 2024-04-19T01:42:22.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d3e5bc640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:22.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d3e5bc640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-19T01:42:22.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d3e5bc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d380a29a0 0x7f4d3813f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:22.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d3e5bc640 10 monclient(hunting): picked mon.noname-a con 0x7f4d380a29a0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:22.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d3e5bc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d380a2dd0 0x7f4d3813f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:22.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d3e5bc640 10 monclient(hunting): picked mon.noname-b con 0x7f4d380a2dd0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:22.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d3e5bc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d380a3c60 0x7f4d38131e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:22.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d3e5bc640 10 monclient(hunting): picked mon.noname-c con 0x7f4d380a3c60 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:22.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d3e5bc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:22.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d3e5bc640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4d38075b50 con 0x7f4d380a29a0 2024-04-19T01:42:22.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d3e5bc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:22.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d3e5bc640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4d380759d0 con 0x7f4d380a2dd0 2024-04-19T01:42:22.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d3e5bc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:22.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d3e5bc640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f4d381435b0 con 0x7f4d380a3c60 2024-04-19T01:42:22.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d3e5bc640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:22.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d3e5bc640 10 monclient(hunting): authenticate will time out at 928.356929s 2024-04-19T01:42:22.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d37fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d380a29a0 0x7f4d3813f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:22.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d37fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d380a29a0 0x7f4d3813f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52410/0 (socket says 172.21.15.62:52410) 2024-04-19T01:42:22.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d37fff640 1 -- 172.21.15.62:0/208640530 learned_addr learned my addr 172.21.15.62:0/208640530 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:22.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d37fff640 10 monclient(hunting): get_auth_request con 0x7f4d380a29a0 auth_method 0 2024-04-19T01:42:22.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:22.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d37fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:22.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d37fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:22.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d3cb59640 1 --2- 172.21.15.62:0/208640530 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d380a3c60 0x7f4d38131e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:22.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d3cb59640 10 monclient(hunting): get_auth_request con 0x7f4d380a3c60 auth_method 0 2024-04-19T01:42:22.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d3cb59640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:22.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d3cb59640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:22.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d3cb59640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:22.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:22.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:22.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:22.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d3cb59640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:22.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d377fe640 1 --2- 172.21.15.62:0/208640530 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d380a2dd0 0x7f4d3813f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:22.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d3cb59640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:22.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d3cb59640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:22.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d377fe640 10 monclient(hunting): get_auth_request con 0x7f4d380a2dd0 auth_method 0 2024-04-19T01:42:22.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:22.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d377fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:22.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d377fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:22.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d37fff640 10 monclient(hunting): handle_auth_done global_id 4185 payload 824 2024-04-19T01:42:22.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d37fff640 1 -- 172.21.15.62:0/208640530 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d380a3c60 msgr2=0x7f4d38131e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:22.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d37fff640 1 --2- 172.21.15.62:0/208640530 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d380a3c60 0x7f4d38131e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:22.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d37fff640 1 -- 172.21.15.62:0/208640530 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d380a2dd0 msgr2=0x7f4d3813f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:22.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d37fff640 1 --2- 172.21.15.62:0/208640530 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d380a2dd0 0x7f4d3813f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:22.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d37fff640 10 monclient: _finish_hunting 0 2024-04-19T01:42:22.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d37fff640 1 monclient: found mon.noname-a 2024-04-19T01:42:22.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:22.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d37fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:22.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d37fff640 1 -- 172.21.15.62:0/208640530 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d38135ea0 con 0x7f4d380a29a0 2024-04-19T01:42:22.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d377fe640 1 --2- 172.21.15.62:0/208640530 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d380a2dd0 0x7f4d3813f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T01:42:22.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d37fff640 1 -- 172.21.15.62:0/208640530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d380a29a0 msgr2=0x7f4d3813f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:22.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d37fff640 1 --2- 172.21.15.62:0/208640530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d380a29a0 0x7f4d3813f430 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c01ccc0 tx=0x7f4d2c0191c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:22.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.534+0000 7f4d34ff9640 1 -- 172.21.15.62:0/208640530 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c0b9020 con 0x7f4d380a29a0 2024-04-19T01:42:22.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.538+0000 7f4d34ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:22.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.538+0000 7f4d34ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:22.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.538+0000 7f4d34ff9640 10 monclient: dump: 2024-04-19T01:42:22.548 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:22.548 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:22.548 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:22.548 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:22.548 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:22.548 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:22.548 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:22.548 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:22.548 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:22.548 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:22.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.538+0000 7f4d34ff9640 10 monclient: _finish_auth 0 2024-04-19T01:42:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.538+0000 7f4d34ff9640 10 monclient: _check_auth_tickets 2024-04-19T01:42:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.538+0000 7f4d34ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.538+0000 7f4d34ff9640 1 -- 172.21.15.62:0/208640530 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d2c0b1040 con 0x7f4d380a29a0 2024-04-19T01:42:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.538+0000 7f4d34ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.538+0000 7f4d34ff9640 1 -- 172.21.15.62:0/208640530 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c0b0050 con 0x7f4d380a29a0 2024-04-19T01:42:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.538+0000 7f4d34ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.538+0000 7f4d34ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.538+0000 7f4d34ff9640 10 monclient: dump: 2024-04-19T01:42:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:22.550 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:22.550 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:22.550 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:22.550 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:22.550 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:22.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.538+0000 7f4d367fc640 4 set_mon_vals no callback set 2024-04-19T01:42:22.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.538+0000 7f4d3e5bc640 5 monclient: authenticate success, global_id 4185 2024-04-19T01:42:22.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.538+0000 7f4d3e5bc640 10 monclient: _renew_subs 2024-04-19T01:42:22.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.538+0000 7f4d3e5bc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:22.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.538+0000 7f4d3e5bc640 1 -- 172.21.15.62:0/208640530 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d38133990 con 0x7f4d380a29a0 2024-04-19T01:42:22.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.538+0000 7f4d3e5bc640 10 monclient: _renew_subs 2024-04-19T01:42:22.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.538+0000 7f4d3e5bc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:22.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.538+0000 7f4d3e5bc640 1 -- 172.21.15.62:0/208640530 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d38135bc0 con 0x7f4d380a29a0 2024-04-19T01:42:22.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.538+0000 7f4d3e5bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:22.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.538+0000 7f4d3e5bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:22.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.538+0000 7f4d3e5bc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:22.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.538+0000 7f4d3e5bc640 1 -- 172.21.15.62:0/208640530 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d38135630 con 0x7f4d380a29a0 2024-04-19T01:42:22.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.538+0000 7f4d34ff9640 1 -- 172.21.15.62:0/208640530 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 4) ==== 44268+0+0 (secure 0 0 0) 0x7f4d2c0bc020 con 0x7f4d380a29a0 2024-04-19T01:42:22.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.538+0000 7f4d34ff9640 1 --2- 172.21.15.62:0/208640530 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4d24036390 0x7f4d24038850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:22.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.538+0000 7f4d34ff9640 1 -- 172.21.15.62:0/208640530 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 3993+0+0 (secure 0 0 0) 0x7f4d2c0e2f40 con 0x7f4d380a29a0 2024-04-19T01:42:22.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.538+0000 7f4d377fe640 1 --2- 172.21.15.62:0/208640530 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4d24036390 0x7f4d24038850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:22.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.538+0000 7f4d377fe640 10 monclient: get_auth_request con 0x7f4d24036390 auth_method 0 2024-04-19T01:42:22.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.542+0000 7f4d377fe640 1 --2- 172.21.15.62:0/208640530 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4d24036390 0x7f4d24038850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4d38133550 tx=0x7f4d28008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:22.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.542+0000 7f4d34ff9640 1 -- 172.21.15.62:0/208640530 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4d2c0c3030 con 0x7f4d380a29a0 2024-04-19T01:42:22.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.542+0000 7f4d34ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:22.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.542+0000 7f4d34ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:22.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.710+0000 7f4d3e5bc640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-04-19T01:42:22.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.710+0000 7f4d3e5bc640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-04-19T01:42:22.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.710+0000 7f4d3e5bc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:22.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:22.710+0000 7f4d3e5bc640 1 -- 172.21.15.62:0/208640530 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7f4d38068ab0 con 0x7f4d380a29a0 2024-04-19T01:42:23.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.114+0000 7f4d34ff9640 1 -- 172.21.15.62:0/208640530 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v14) ==== 109+0+0 (secure 0 0 0) 0x7f4d2c016130 con 0x7f4d380a29a0 2024-04-19T01:42:23.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.114+0000 7f4d34ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-04-19T01:42:23.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.114+0000 7f4d34ff9640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-04-19T01:42:23.117 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'rbd' created 2024-04-19T01:42:23.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.118+0000 7f4d3e5bc640 1 -- 172.21.15.62:0/208640530 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4d24036390 msgr2=0x7f4d24038850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:23.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.118+0000 7f4d3e5bc640 1 --2- 172.21.15.62:0/208640530 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4d24036390 0x7f4d24038850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4d38133550 tx=0x7f4d28008040 comp rx=0 tx=0).stop 2024-04-19T01:42:23.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.118+0000 7f4d3e5bc640 10 monclient: shutdown 2024-04-19T01:42:23.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.118+0000 7f4d3e5bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:23.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.118+0000 7f4d3e5bc640 1 -- 172.21.15.62:0/208640530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d380a29a0 msgr2=0x7f4d3813f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:23.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.118+0000 7f4d3e5bc640 1 --2- 172.21.15.62:0/208640530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d380a29a0 0x7f4d3813f430 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c01ccc0 tx=0x7f4d2c0191c0 comp rx=0 tx=0).stop 2024-04-19T01:42:23.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.118+0000 7f4d3e5bc640 1 -- 172.21.15.62:0/208640530 shutdown_connections 2024-04-19T01:42:23.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.118+0000 7f4d3e5bc640 1 --2- 172.21.15.62:0/208640530 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4d24036390 0x7f4d24038850 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:23.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.118+0000 7f4d3e5bc640 1 --2- 172.21.15.62:0/208640530 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d380a3c60 0x7f4d38131e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:23.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.118+0000 7f4d3e5bc640 1 --2- 172.21.15.62:0/208640530 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d380a2dd0 0x7f4d3813f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:23.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.118+0000 7f4d3e5bc640 1 --2- 172.21.15.62:0/208640530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d380a29a0 0x7f4d3813f430 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:23.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.118+0000 7f4d3e5bc640 1 -- 172.21.15.62:0/208640530 >> 172.21.15.62:0/208640530 conn(0x7f4d380a7050 msgr2=0x7f4d3812fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:23.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.118+0000 7f4d3e5bc640 1 -- 172.21.15.62:0/208640530 shutdown_connections 2024-04-19T01:42:23.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.118+0000 7f4d3e5bc640 1 -- 172.21.15.62:0/208640530 wait complete. 2024-04-19T01:42:23.145 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-19T01:42:23.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.262+0000 7f427c487640 10 monclient: get_monmap_and_config 2024-04-19T01:42:23.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.262+0000 7f427c487640 10 monclient: build_initial_monmap 2024-04-19T01:42:23.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.262+0000 7f427c487640 10 monclient: monmap: 2024-04-19T01:42:23.265 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:23.265 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:23.265 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:23.268962+0000 2024-04-19T01:42:23.265 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:23.268962+0000 2024-04-19T01:42:23.265 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:23.265 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:23.265 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:23.265 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:23.265 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:23.265 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:23.265 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:23.266 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:23.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.262+0000 7f427c487640 1 Processor -- start 2024-04-19T01:42:23.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.262+0000 7f427c487640 1 -- start start 2024-04-19T01:42:23.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.262+0000 7f427c487640 10 monclient: init 2024-04-19T01:42:23.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427c487640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:23.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427c487640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-04-19T01:42:23.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427c487640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4274098570 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:23.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427c487640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f427409a270 0x7f4274138a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:23.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427c487640 10 monclient(hunting): picked mon.noname-c con 0x7f427409a270 addr v2:172.21.15.71:3300/0 2024-04-19T01:42:23.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427c487640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4274138fd0 0x7f427413b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:23.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427c487640 10 monclient(hunting): picked mon.noname-a con 0x7f4274138fd0 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:23.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427c487640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4274099060 con 0x7f4274098570 2024-04-19T01:42:23.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427c487640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:23.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427c487640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f427409a680 con 0x7f4274138fd0 2024-04-19T01:42:23.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427c487640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:23.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427c487640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f427409dde0 con 0x7f427409a270 2024-04-19T01:42:23.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427c487640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:23.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427c487640 10 monclient(hunting): authenticate will time out at 929.088509s 2024-04-19T01:42:23.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427a223640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f4274098570 0x7f4274098950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:60986/0 (socket says 172.21.15.62:60986) 2024-04-19T01:42:23.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427a223640 1 -- 172.21.15.62:0/2469899646 learned_addr learned my addr 172.21.15.62:0/2469899646 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:42:23.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427aa24640 1 --2- 172.21.15.62:0/2469899646 >> v2:172.21.15.62:3300/0 conn(0x7f4274138fd0 0x7f427413b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:23.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427aa24640 10 monclient(hunting): get_auth_request con 0x7f4274138fd0 auth_method 0 2024-04-19T01:42:23.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427aa24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:23.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427aa24640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:23.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427aa24640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:23.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f4279a22640 1 --2- 172.21.15.62:0/2469899646 >> v2:172.21.15.71:3300/0 conn(0x7f427409a270 0x7f4274138a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:23.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427aa24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:23.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427aa24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:23.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427aa24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:23.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f4279a22640 10 monclient(hunting): get_auth_request con 0x7f427409a270 auth_method 0 2024-04-19T01:42:23.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f4279a22640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:23.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f4279a22640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:23.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f4279a22640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:23.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427aa24640 10 monclient(hunting): handle_auth_done global_id 4191 payload 274 2024-04-19T01:42:23.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427aa24640 1 -- 172.21.15.62:0/2469899646 >> v2:172.21.15.71:3300/0 conn(0x7f427409a270 msgr2=0x7f4274138a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:23.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427aa24640 1 --2- 172.21.15.62:0/2469899646 >> v2:172.21.15.71:3300/0 conn(0x7f427409a270 0x7f4274138a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:23.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427aa24640 1 -- 172.21.15.62:0/2469899646 >> v1:172.21.15.62:6789/0 conn(0x7f4274098570 legacy=0x7f4274098950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:23.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427aa24640 10 monclient: _finish_hunting 0 2024-04-19T01:42:23.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427aa24640 1 monclient: found mon.noname-a 2024-04-19T01:42:23.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427aa24640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:23.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427aa24640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:23.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.266+0000 7f427aa24640 1 -- 172.21.15.62:0/2469899646 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f427409e020 con 0x7f4274138fd0 2024-04-19T01:42:23.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427aa24640 1 -- 172.21.15.62:0/2469899646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4274138fd0 msgr2=0x7f427413b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427aa24640 1 --2- 172.21.15.62:0/2469899646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4274138fd0 0x7f427413b3c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4274099350 tx=0x7f426c01c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=458a594a65affb51 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f4279221640 1 -- 172.21.15.62:0/2469899646 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426c0b2070 con 0x7f4274138fd0 2024-04-19T01:42:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f4279221640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f4279221640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f4279221640 10 monclient: dump: 2024-04-19T01:42:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:23.278 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:23.278 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:23.278 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:23.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f4279221640 10 monclient: _finish_auth 0 2024-04-19T01:42:23.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f4279221640 10 monclient: _check_auth_tickets 2024-04-19T01:42:23.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f4279221640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:23.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f4279221640 1 -- 172.21.15.62:0/2469899646 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f426c0a5790 con 0x7f4274138fd0 2024-04-19T01:42:23.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 5 monclient: authenticate success, global_id 4191 2024-04-19T01:42:23.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:42:23.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f4279221640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:23.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:23.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 4 set_mon_vals no callback set 2024-04-19T01:42:23.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f4279221640 1 -- 172.21.15.62:0/2469899646 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426c0a5b60 con 0x7f4274138fd0 2024-04-19T01:42:23.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f4279221640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:23.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f4279221640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:23.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f4279221640 10 monclient: dump: 2024-04-19T01:42:23.279 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:23.279 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:23.279 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:23.279 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:23.279 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:23.279 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:23.279 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:23.279 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:23.280 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:23.280 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:23.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 10 monclient: shutdown 2024-04-19T01:42:23.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:23.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 1 -- 172.21.15.62:0/2469899646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4274138fd0 msgr2=0x7f427413b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:23.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 1 --2- 172.21.15.62:0/2469899646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4274138fd0 0x7f427413b3c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4274099350 tx=0x7f426c01c880 comp rx=0 tx=0).stop 2024-04-19T01:42:23.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 1 -- 172.21.15.62:0/2469899646 shutdown_connections 2024-04-19T01:42:23.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 1 --2- 172.21.15.62:0/2469899646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4274138fd0 0x7f427413b3c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:23.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 1 --2- 172.21.15.62:0/2469899646 >> v2:172.21.15.71:3300/0 conn(0x7f427409a270 0x7f4274138a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:23.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 1 -- 172.21.15.62:0/2469899646 >> 172.21.15.62:0/2469899646 conn(0x7f42740a2190 msgr2=0x7f42740a25a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:23.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 1 -- 172.21.15.62:0/2469899646 shutdown_connections 2024-04-19T01:42:23.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 1 -- 172.21.15.62:0/2469899646 wait complete. 2024-04-19T01:42:23.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 10 monclient: build_initial_monmap 2024-04-19T01:42:23.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 10 monclient: monmap: 2024-04-19T01:42:23.281 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:23.281 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:23.281 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:23.281 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:23.281 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:23.281 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:23.281 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:23.281 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:23.281 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:23.281 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:23.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 1 Processor -- start 2024-04-19T01:42:23.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 1 -- start start 2024-04-19T01:42:23.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 10 monclient: init 2024-04-19T01:42:23.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:23.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-19T01:42:23.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4274098570 0x7f4274071670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:23.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 10 monclient(hunting): picked mon.noname-a con 0x7f4274098570 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:23.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f427409a270 0x7f4274071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:23.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 10 monclient(hunting): picked mon.noname-b con 0x7f427409a270 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:23.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4274138fd0 0x7f42740720f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:23.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 10 monclient(hunting): picked mon.noname-c con 0x7f4274138fd0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:23.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:23.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f427413ba80 con 0x7f4274098570 2024-04-19T01:42:23.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:23.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f427413b900 con 0x7f427409a270 2024-04-19T01:42:23.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:23.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f427413bc00 con 0x7f4274138fd0 2024-04-19T01:42:23.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:23.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427c487640 10 monclient(hunting): authenticate will time out at 929.095787s 2024-04-19T01:42:23.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427a223640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4274098570 0x7f4274071670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:23.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427a223640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4274098570 0x7f4274071670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52430/0 (socket says 172.21.15.62:52430) 2024-04-19T01:42:23.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427a223640 1 -- 172.21.15.62:0/3159267734 learned_addr learned my addr 172.21.15.62:0/3159267734 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:23.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427a223640 10 monclient(hunting): get_auth_request con 0x7f4274098570 auth_method 0 2024-04-19T01:42:23.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427a223640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:23.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427a223640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:23.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427a223640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:23.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427aa24640 1 --2- 172.21.15.62:0/3159267734 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4274138fd0 0x7f42740720f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:23.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427aa24640 10 monclient(hunting): get_auth_request con 0x7f4274138fd0 auth_method 0 2024-04-19T01:42:23.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427aa24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:23.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427aa24640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:23.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.270+0000 7f427aa24640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:23.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f427a223640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:23.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f427a223640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:23.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f427a223640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:23.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f427aa24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:23.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f427aa24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:23.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f427aa24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:23.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f4279a22640 1 --2- 172.21.15.62:0/3159267734 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f427409a270 0x7f4274071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:23.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f4279a22640 10 monclient(hunting): get_auth_request con 0x7f427409a270 auth_method 0 2024-04-19T01:42:23.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f4279a22640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:23.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f4279a22640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:23.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f4279a22640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:23.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f427a223640 10 monclient(hunting): handle_auth_done global_id 4194 payload 824 2024-04-19T01:42:23.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f427a223640 1 -- 172.21.15.62:0/3159267734 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4274138fd0 msgr2=0x7f42740720f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:23.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f427a223640 1 --2- 172.21.15.62:0/3159267734 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4274138fd0 0x7f42740720f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:23.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f427a223640 1 -- 172.21.15.62:0/3159267734 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f427409a270 msgr2=0x7f4274071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:23.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f427a223640 1 --2- 172.21.15.62:0/3159267734 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f427409a270 0x7f4274071bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:23.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f427a223640 10 monclient: _finish_hunting 0 2024-04-19T01:42:23.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f427a223640 1 monclient: found mon.noname-a 2024-04-19T01:42:23.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f427a223640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:23.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f427a223640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:23.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f427a223640 1 -- 172.21.15.62:0/3159267734 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42740730b0 con 0x7f4274098570 2024-04-19T01:42:23.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f427a223640 1 -- 172.21.15.62:0/3159267734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4274098570 msgr2=0x7f4274071670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:23.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f427a223640 1 --2- 172.21.15.62:0/3159267734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4274098570 0x7f4274071670 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f427000c470 tx=0x7f427000c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:23.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f426affd640 1 -- 172.21.15.62:0/3159267734 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4270007070 con 0x7f4274098570 2024-04-19T01:42:23.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f426affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:23.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f426affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:23.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f426affd640 10 monclient: dump: 2024-04-19T01:42:23.286 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:23.286 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:23.286 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:23.287 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:23.287 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:23.287 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:23.287 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:23.287 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:23.287 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:23.287 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:23.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f426affd640 10 monclient: _finish_auth 0 2024-04-19T01:42:23.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f426affd640 10 monclient: _check_auth_tickets 2024-04-19T01:42:23.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f426affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:23.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f426affd640 1 -- 172.21.15.62:0/3159267734 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4270007210 con 0x7f4274098570 2024-04-19T01:42:23.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f426affd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:23.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f426affd640 1 -- 172.21.15.62:0/3159267734 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4270007600 con 0x7f4274098570 2024-04-19T01:42:23.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f426affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:23.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f426affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:23.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f426affd640 10 monclient: dump: 2024-04-19T01:42:23.288 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:23.288 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:23.288 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:23.288 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:23.288 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:23.288 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:23.288 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:23.288 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:23.288 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:23.288 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:23.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f427c487640 5 monclient: authenticate success, global_id 4194 2024-04-19T01:42:23.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f427c487640 10 monclient: _renew_subs 2024-04-19T01:42:23.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f427c487640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:23.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f426bfff640 4 set_mon_vals no callback set 2024-04-19T01:42:23.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f427c487640 1 -- 172.21.15.62:0/3159267734 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f427408e1a0 con 0x7f4274098570 2024-04-19T01:42:23.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f427c487640 10 monclient: _renew_subs 2024-04-19T01:42:23.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f427c487640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:23.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f427c487640 1 -- 172.21.15.62:0/3159267734 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f427408d000 con 0x7f4274098570 2024-04-19T01:42:23.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f427c487640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:23.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f427c487640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:23.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f427c487640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:23.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f427c487640 1 -- 172.21.15.62:0/3159267734 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f427408e8e0 con 0x7f4274098570 2024-04-19T01:42:23.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.274+0000 7f426affd640 1 -- 172.21.15.62:0/3159267734 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 4) ==== 44268+0+0 (secure 0 0 0) 0x7f427001b400 con 0x7f4274098570 2024-04-19T01:42:23.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.278+0000 7f426affd640 1 --2- 172.21.15.62:0/3159267734 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f42580363e0 0x7f42580388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:23.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.278+0000 7f426affd640 1 -- 172.21.15.62:0/3159267734 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 4335+0+0 (secure 0 0 0) 0x7f4270048ee0 con 0x7f4274098570 2024-04-19T01:42:23.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.278+0000 7f4279a22640 1 --2- 172.21.15.62:0/3159267734 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f42580363e0 0x7f42580388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:23.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.278+0000 7f4279a22640 10 monclient: get_auth_request con 0x7f42580363e0 auth_method 0 2024-04-19T01:42:23.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.278+0000 7f4279a22640 1 --2- 172.21.15.62:0/3159267734 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f42580363e0 0x7f42580388a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f42741389e0 tx=0x7f4264008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:23.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.282+0000 7f426affd640 1 -- 172.21.15.62:0/3159267734 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f427001b980 con 0x7f4274098570 2024-04-19T01:42:23.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.282+0000 7f426affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:23.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.282+0000 7f426affd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:23.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.446+0000 7f427c487640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-04-19T01:42:23.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.446+0000 7f427c487640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-04-19T01:42:23.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.446+0000 7f427c487640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:23.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.446+0000 7f427c487640 1 -- 172.21.15.62:0/3159267734 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) -- 0x7f427404fcf0 con 0x7f4274098570 2024-04-19T01:42:23.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:23.458+0000 7f426affd640 1 -- 172.21.15.62:0/3159267734 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f4270012400 con 0x7f4274098570 2024-04-19T01:42:24.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.118+0000 7f426affd640 1 -- 172.21.15.62:0/3159267734 <== mon.0 v2:172.21.15.62:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v15) ==== 173+0+0 (secure 0 0 0) 0x7f427000fd80 con 0x7f4274098570 2024-04-19T01:42:24.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.118+0000 7f426affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-04-19T01:42:24.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.118+0000 7f426affd640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2024-04-19T01:42:24.196 INFO:teuthology.orchestra.run.smithi062.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-19T01:42:24.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.126+0000 7f427c487640 1 -- 172.21.15.62:0/3159267734 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f42580363e0 msgr2=0x7f42580388a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:24.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.126+0000 7f427c487640 1 --2- 172.21.15.62:0/3159267734 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f42580363e0 0x7f42580388a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f42741389e0 tx=0x7f4264008040 comp rx=0 tx=0).stop 2024-04-19T01:42:24.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.126+0000 7f427c487640 10 monclient: shutdown 2024-04-19T01:42:24.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.126+0000 7f427c487640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:24.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.126+0000 7f427c487640 1 -- 172.21.15.62:0/3159267734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4274098570 msgr2=0x7f4274071670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:24.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.126+0000 7f427c487640 1 --2- 172.21.15.62:0/3159267734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4274098570 0x7f4274071670 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f427000c470 tx=0x7f427000c930 comp rx=0 tx=0).stop 2024-04-19T01:42:24.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.126+0000 7f427c487640 1 -- 172.21.15.62:0/3159267734 shutdown_connections 2024-04-19T01:42:24.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.126+0000 7f427c487640 1 --2- 172.21.15.62:0/3159267734 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f42580363e0 0x7f42580388a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:24.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.126+0000 7f427c487640 1 --2- 172.21.15.62:0/3159267734 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4274138fd0 0x7f42740720f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:24.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.126+0000 7f427c487640 1 --2- 172.21.15.62:0/3159267734 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f427409a270 0x7f4274071bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:24.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.126+0000 7f427c487640 1 --2- 172.21.15.62:0/3159267734 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4274098570 0x7f4274071670 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:24.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.126+0000 7f427c487640 1 -- 172.21.15.62:0/3159267734 >> 172.21.15.62:0/3159267734 conn(0x7f42740a2190 msgr2=0x7f427413c910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:24.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.126+0000 7f427c487640 1 -- 172.21.15.62:0/3159267734 shutdown_connections 2024-04-19T01:42:24.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.126+0000 7f427c487640 1 -- 172.21.15.62:0/3159267734 wait complete. 2024-04-19T01:42:24.199 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-19T01:42:24.199 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-19T01:42:24.199 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-19T01:42:24.199 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-19T01:42:24.200 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-19T01:42:24.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.322+0000 7fd2f037c640 10 monclient: get_monmap_and_config 2024-04-19T01:42:24.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.322+0000 7fd2f037c640 10 monclient: build_initial_monmap 2024-04-19T01:42:24.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.322+0000 7fd2f037c640 10 monclient: monmap: 2024-04-19T01:42:24.325 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:24.325 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:24.325 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:24.329212+0000 2024-04-19T01:42:24.325 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:24.329212+0000 2024-04-19T01:42:24.325 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:24.325 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:24.325 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:24.325 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:24.325 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:24.325 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:24.325 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:24.326 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:24.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2f037c640 1 Processor -- start 2024-04-19T01:42:24.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2f037c640 1 -- start start 2024-04-19T01:42:24.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2f037c640 10 monclient: init 2024-04-19T01:42:24.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2f037c640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:24.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2f037c640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-04-19T01:42:24.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2f037c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd2e80a47e0 0x7fd2e809f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:24.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2f037c640 10 monclient(hunting): picked mon.noname-c con 0x7fd2e80a47e0 addr v2:172.21.15.71:3300/0 2024-04-19T01:42:24.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2f037c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2e809f890 0x7fd2e80a4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:24.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2f037c640 10 monclient(hunting): picked mon.noname-b con 0x7fd2e809f890 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:24.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2f037c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd2e80a5100 0x7fd2e81340a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:24.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2f037c640 10 monclient(hunting): picked mon.noname-a con 0x7fd2e80a5100 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:24.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2f037c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:24.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2f037c640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fd2e809ef30 con 0x7fd2e80a5100 2024-04-19T01:42:24.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2f037c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:24.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2f037c640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fd2e809e690 con 0x7fd2e80a47e0 2024-04-19T01:42:24.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2f037c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:24.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2f037c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fd2e809ec50 con 0x7fd2e809f890 2024-04-19T01:42:24.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2f037c640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:24.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2f037c640 10 monclient(hunting): authenticate will time out at 930.148981s 2024-04-19T01:42:24.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ee919640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd2e80a5100 0x7fd2e81340a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:24.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ed917640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2e809f890 0x7fd2e80a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:24.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ed917640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2e809f890 0x7fd2e80a4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:57782/0 (socket says 172.21.15.62:57782) 2024-04-19T01:42:24.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ee919640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd2e80a5100 0x7fd2e81340a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52440/0 (socket says 172.21.15.62:52440) 2024-04-19T01:42:24.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ed917640 1 -- 172.21.15.62:0/611676170 learned_addr learned my addr 172.21.15.62:0/611676170 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:24.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ed917640 10 monclient(hunting): get_auth_request con 0x7fd2e809f890 auth_method 0 2024-04-19T01:42:24.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ed917640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:24.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ed917640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:24.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ed917640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:24.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ee919640 10 monclient(hunting): get_auth_request con 0x7fd2e80a5100 auth_method 0 2024-04-19T01:42:24.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ee919640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:24.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ee919640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:24.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ee919640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:24.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ed917640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:24.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ed917640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:24.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ed917640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:24.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ee118640 1 --2- 172.21.15.62:0/611676170 >> v2:172.21.15.71:3300/0 conn(0x7fd2e80a47e0 0x7fd2e809f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:24.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ee919640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:24.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ee919640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:24.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ee919640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:24.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ee118640 10 monclient(hunting): get_auth_request con 0x7fd2e80a47e0 auth_method 0 2024-04-19T01:42:24.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ee118640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:24.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ee118640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:24.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ee118640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:24.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ed917640 10 monclient(hunting): handle_auth_done global_id 4202 payload 274 2024-04-19T01:42:24.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ed917640 1 -- 172.21.15.62:0/611676170 >> v2:172.21.15.71:3300/0 conn(0x7fd2e80a47e0 msgr2=0x7fd2e809f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:24.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ed917640 1 --2- 172.21.15.62:0/611676170 >> v2:172.21.15.71:3300/0 conn(0x7fd2e80a47e0 0x7fd2e809f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:24.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ed917640 1 -- 172.21.15.62:0/611676170 >> v2:172.21.15.62:3300/0 conn(0x7fd2e80a5100 msgr2=0x7fd2e81340a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:24.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ed917640 1 --2- 172.21.15.62:0/611676170 >> v2:172.21.15.62:3300/0 conn(0x7fd2e80a5100 0x7fd2e81340a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:24.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ed917640 10 monclient: _finish_hunting 0 2024-04-19T01:42:24.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ed917640 1 monclient: found mon.noname-b 2024-04-19T01:42:24.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ed917640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:24.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ed917640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:42:24.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ed917640 1 -- 172.21.15.62:0/611676170 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2e809e970 con 0x7fd2e809f890 2024-04-19T01:42:24.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ee118640 1 --2- 172.21.15.62:0/611676170 >> v2:172.21.15.71:3300/0 conn(0x7fd2e80a47e0 0x7fd2e809f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T01:42:24.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.326+0000 7fd2ee919640 1 --2- 172.21.15.62:0/611676170 >> v2:172.21.15.62:3300/0 conn(0x7fd2e80a5100 0x7fd2e81340a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:42:24.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2ed917640 1 -- 172.21.15.62:0/611676170 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2e809f890 msgr2=0x7fd2e80a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:24.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2ed917640 1 --2- 172.21.15.62:0/611676170 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2e809f890 0x7fd2e80a4bc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd2e809d3b0 tx=0x7fd2e401c1b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2414e3799e7195ee server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:24.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2ed116640 1 -- 172.21.15.62:0/611676170 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e40b0070 con 0x7fd2e809f890 2024-04-19T01:42:24.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2ed116640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:24.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2ed116640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:42:24.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2ed116640 10 monclient: dump: 2024-04-19T01:42:24.336 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:24.336 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:24.336 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:24.336 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:24.336 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:24.336 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:24.336 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:24.336 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:24.336 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:24.336 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:24.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2ed116640 10 monclient: _finish_auth 0 2024-04-19T01:42:24.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2ed116640 10 monclient: _check_auth_tickets 2024-04-19T01:42:24.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2ed116640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:24.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2ed116640 1 -- 172.21.15.62:0/611676170 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2e40ab600 con 0x7fd2e809f890 2024-04-19T01:42:24.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2f037c640 5 monclient: authenticate success, global_id 4202 2024-04-19T01:42:24.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2f037c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:42:24.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2ed116640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:24.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2f037c640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:24.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2ed116640 1 -- 172.21.15.62:0/611676170 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e40ab9d0 con 0x7fd2e809f890 2024-04-19T01:42:24.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2f037c640 4 set_mon_vals no callback set 2024-04-19T01:42:24.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2ed116640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:24.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2ed116640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:24.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2ed116640 10 monclient: dump: 2024-04-19T01:42:24.337 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:24.337 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:24.337 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:24.338 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:24.338 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:24.338 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:24.338 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:24.338 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:24.338 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:24.338 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:24.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2f037c640 10 monclient: shutdown 2024-04-19T01:42:24.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2f037c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:24.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2f037c640 1 -- 172.21.15.62:0/611676170 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2e809f890 msgr2=0x7fd2e80a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:24.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2f037c640 1 --2- 172.21.15.62:0/611676170 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2e809f890 0x7fd2e80a4bc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd2e809d3b0 tx=0x7fd2e401c1b0 comp rx=0 tx=0).stop 2024-04-19T01:42:24.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2f037c640 1 -- 172.21.15.62:0/611676170 shutdown_connections 2024-04-19T01:42:24.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2f037c640 1 --2- 172.21.15.62:0/611676170 >> v2:172.21.15.62:3300/0 conn(0x7fd2e80a5100 0x7fd2e81340a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:24.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2f037c640 1 --2- 172.21.15.62:0/611676170 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2e809f890 0x7fd2e80a4bc0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:24.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2f037c640 1 --2- 172.21.15.62:0/611676170 >> v2:172.21.15.71:3300/0 conn(0x7fd2e80a47e0 0x7fd2e809f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:24.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2f037c640 1 -- 172.21.15.62:0/611676170 >> 172.21.15.62:0/611676170 conn(0x7fd2e80a23d0 msgr2=0x7fd2e80a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:24.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2f037c640 1 -- 172.21.15.62:0/611676170 shutdown_connections 2024-04-19T01:42:24.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2f037c640 1 -- 172.21.15.62:0/611676170 wait complete. 2024-04-19T01:42:24.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2f037c640 10 monclient: build_initial_monmap 2024-04-19T01:42:24.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2f037c640 10 monclient: monmap: 2024-04-19T01:42:24.339 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:24.339 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:24.339 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:24.339 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:24.339 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:24.339 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:24.339 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:24.339 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:24.339 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:24.339 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:24.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2f037c640 1 Processor -- start 2024-04-19T01:42:24.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2f037c640 1 -- start start 2024-04-19T01:42:24.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.330+0000 7fd2f037c640 10 monclient: init 2024-04-19T01:42:24.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2f037c640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:24.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2f037c640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-19T01:42:24.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2f037c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd2e809f890 0x7fd2e813e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:24.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2f037c640 10 monclient(hunting): picked mon.noname-b con 0x7fd2e809f890 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:24.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2f037c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2e80a47e0 0x7fd2e81cf4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:24.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2f037c640 10 monclient(hunting): picked mon.noname-a con 0x7fd2e80a47e0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:24.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2f037c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2e80a5100 0x7fd2e812f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:24.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2f037c640 10 monclient(hunting): picked mon.noname-c con 0x7fd2e80a5100 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:24.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2f037c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:24.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2f037c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e813da50 con 0x7fd2e80a47e0 2024-04-19T01:42:24.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2f037c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:24.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2f037c640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e813d8d0 con 0x7fd2e809f890 2024-04-19T01:42:24.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2f037c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:24.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2f037c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fd2e813dbd0 con 0x7fd2e80a5100 2024-04-19T01:42:24.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2f037c640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:24.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2f037c640 10 monclient(hunting): authenticate will time out at 930.156453s 2024-04-19T01:42:24.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ed917640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2e80a47e0 0x7fd2e81cf4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:24.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ee919640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2e80a5100 0x7fd2e812f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:24.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ed917640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2e80a47e0 0x7fd2e81cf4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52452/0 (socket says 172.21.15.62:52452) 2024-04-19T01:42:24.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ed917640 1 -- 172.21.15.62:0/1527775754 learned_addr learned my addr 172.21.15.62:0/1527775754 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:24.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ee919640 10 monclient(hunting): get_auth_request con 0x7fd2e80a5100 auth_method 0 2024-04-19T01:42:24.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ee919640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:24.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ee919640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:24.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ee919640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:24.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ed917640 10 monclient(hunting): get_auth_request con 0x7fd2e80a47e0 auth_method 0 2024-04-19T01:42:24.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ed917640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:24.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ed917640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:24.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ed917640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:24.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ee919640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:24.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ee919640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:24.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ee919640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:24.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ed917640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:24.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ed917640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:24.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ed917640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:24.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ee118640 1 --2- 172.21.15.62:0/1527775754 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd2e809f890 0x7fd2e813e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:24.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ee118640 10 monclient(hunting): get_auth_request con 0x7fd2e809f890 auth_method 0 2024-04-19T01:42:24.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ee118640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:24.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ee118640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:24.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ee118640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:24.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ee919640 10 monclient(hunting): handle_auth_done global_id 4205 payload 824 2024-04-19T01:42:24.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ee919640 1 -- 172.21.15.62:0/1527775754 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd2e809f890 msgr2=0x7fd2e813e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:24.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ee919640 1 --2- 172.21.15.62:0/1527775754 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd2e809f890 0x7fd2e813e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:24.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ee919640 1 -- 172.21.15.62:0/1527775754 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2e80a47e0 msgr2=0x7fd2e81cf4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:24.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ee919640 1 --2- 172.21.15.62:0/1527775754 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2e80a47e0 0x7fd2e81cf4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:24.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ee919640 10 monclient: _finish_hunting 0 2024-04-19T01:42:24.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ee919640 1 monclient: found mon.noname-c 2024-04-19T01:42:24.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ee919640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:24.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ee919640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:42:24.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ee919640 1 -- 172.21.15.62:0/1527775754 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2e81cfa00 con 0x7fd2e80a5100 2024-04-19T01:42:24.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ee919640 1 -- 172.21.15.62:0/1527775754 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2e80a5100 msgr2=0x7fd2e812f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:24.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ee919640 1 --2- 172.21.15.62:0/1527775754 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2e80a5100 0x7fd2e812f3c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd2e000efc0 tx=0x7fd2e000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:24.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2deffd640 1 -- 172.21.15.62:0/1527775754 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e0018030 con 0x7fd2e80a5100 2024-04-19T01:42:24.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:24.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2deffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:24.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2deffd640 10 monclient: dump: 2024-04-19T01:42:24.345 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:24.345 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:24.345 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:24.346 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:24.346 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:24.346 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:24.346 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:24.346 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:24.346 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:24.346 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:24.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2deffd640 10 monclient: _finish_auth 0 2024-04-19T01:42:24.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2deffd640 10 monclient: _check_auth_tickets 2024-04-19T01:42:24.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:24.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2deffd640 1 -- 172.21.15.62:0/1527775754 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2e00039a0 con 0x7fd2e80a5100 2024-04-19T01:42:24.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2deffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:24.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2f037c640 5 monclient: authenticate success, global_id 4205 2024-04-19T01:42:24.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2deffd640 1 -- 172.21.15.62:0/1527775754 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e0003d70 con 0x7fd2e80a5100 2024-04-19T01:42:24.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2f037c640 10 monclient: _renew_subs 2024-04-19T01:42:24.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2f037c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:24.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2f037c640 1 -- 172.21.15.62:0/1527775754 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2e809e000 con 0x7fd2e80a5100 2024-04-19T01:42:24.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:24.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2ec915640 4 set_mon_vals no callback set 2024-04-19T01:42:24.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:24.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2deffd640 10 monclient: dump: 2024-04-19T01:42:24.347 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:24.347 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:24.347 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:24.347 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:24.347 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:24.347 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:24.347 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:24.347 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:24.348 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:24.348 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:24.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2f037c640 10 monclient: _renew_subs 2024-04-19T01:42:24.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2f037c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:24.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.334+0000 7fd2f037c640 1 -- 172.21.15.62:0/1527775754 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2e813afe0 con 0x7fd2e80a5100 2024-04-19T01:42:24.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.338+0000 7fd2f037c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:24.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.338+0000 7fd2f037c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:24.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.338+0000 7fd2f037c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:24.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.338+0000 7fd2f037c640 1 -- 172.21.15.62:0/1527775754 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2e80692b0 con 0x7fd2e80a5100 2024-04-19T01:42:24.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.338+0000 7fd2deffd640 1 -- 172.21.15.62:0/1527775754 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fd2e001b070 con 0x7fd2e80a5100 2024-04-19T01:42:24.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.338+0000 7fd2deffd640 1 --2- 172.21.15.62:0/1527775754 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fd2cc036470 0x7fd2cc038930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:24.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.338+0000 7fd2deffd640 1 -- 172.21.15.62:0/1527775754 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7fd2e0049be0 con 0x7fd2e80a5100 2024-04-19T01:42:24.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.338+0000 7fd2ee118640 1 --2- 172.21.15.62:0/1527775754 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fd2cc036470 0x7fd2cc038930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:24.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.338+0000 7fd2ee118640 10 monclient: get_auth_request con 0x7fd2cc036470 auth_method 0 2024-04-19T01:42:24.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.342+0000 7fd2ee118640 1 --2- 172.21.15.62:0/1527775754 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fd2cc036470 0x7fd2cc038930 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd2d80098b0 tx=0x7fd2d8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:24.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.342+0000 7fd2deffd640 1 -- 172.21.15.62:0/1527775754 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd2e0022070 con 0x7fd2e80a5100 2024-04-19T01:42:24.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.342+0000 7fd2deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:24.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.342+0000 7fd2deffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:24.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.546+0000 7fd2f037c640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-04-19T01:42:24.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.546+0000 7fd2f037c640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-04-19T01:42:24.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.546+0000 7fd2f037c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:24.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.546+0000 7fd2f037c640 1 -- 172.21.15.62:0/1527775754 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fd2e806b980 con 0x7fd2e80a5100 2024-04-19T01:42:24.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.550+0000 7fd2deffd640 1 -- 172.21.15.62:0/1527775754 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+84772 (secure 0 0 0) 0x7fd2e0010040 con 0x7fd2e80a5100 2024-04-19T01:42:24.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.550+0000 7fd2deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-04-19T01:42:24.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.550+0000 7fd2deffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:24.553 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:42:24.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.554+0000 7fd2f037c640 1 -- 172.21.15.62:0/1527775754 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fd2cc036470 msgr2=0x7fd2cc038930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:24.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.554+0000 7fd2f037c640 1 --2- 172.21.15.62:0/1527775754 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fd2cc036470 0x7fd2cc038930 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd2d80098b0 tx=0x7fd2d8008040 comp rx=0 tx=0).stop 2024-04-19T01:42:24.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.554+0000 7fd2f037c640 10 monclient: shutdown 2024-04-19T01:42:24.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.554+0000 7fd2f037c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:24.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.554+0000 7fd2f037c640 1 -- 172.21.15.62:0/1527775754 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2e80a5100 msgr2=0x7fd2e812f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:24.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.554+0000 7fd2f037c640 1 --2- 172.21.15.62:0/1527775754 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2e80a5100 0x7fd2e812f3c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd2e000efc0 tx=0x7fd2e000c490 comp rx=0 tx=0).stop 2024-04-19T01:42:24.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.554+0000 7fd2f037c640 1 -- 172.21.15.62:0/1527775754 shutdown_connections 2024-04-19T01:42:24.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.554+0000 7fd2f037c640 1 --2- 172.21.15.62:0/1527775754 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fd2cc036470 0x7fd2cc038930 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:24.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.554+0000 7fd2f037c640 1 --2- 172.21.15.62:0/1527775754 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd2e80a5100 0x7fd2e812f3c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:24.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.554+0000 7fd2f037c640 1 --2- 172.21.15.62:0/1527775754 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2e80a47e0 0x7fd2e81cf4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:24.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.554+0000 7fd2f037c640 1 --2- 172.21.15.62:0/1527775754 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd2e809f890 0x7fd2e813e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:24.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.554+0000 7fd2f037c640 1 -- 172.21.15.62:0/1527775754 >> 172.21.15.62:0/1527775754 conn(0x7fd2e80a23d0 msgr2=0x7fd2e80a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:24.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.554+0000 7fd2f037c640 1 -- 172.21.15.62:0/1527775754 shutdown_connections 2024-04-19T01:42:24.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.554+0000 7fd2f037c640 1 -- 172.21.15.62:0/1527775754 wait complete. 2024-04-19T01:42:24.572 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":5,"flags":0,"active_gid":4108,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":2133533992},{"type":"v1","addr":"172.21.15.71:6825","nonce":2133533992}]},"active_addr":"172.21.15.71:6825/2133533992","active_change":"2024-04-19T01:42:14.690673+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.71:0","nonce":146190020}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.71:0","nonce":3184914478}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.71:0","nonce":3447198372}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.71:0","nonce":2387837327}]}]} 2024-04-19T01:42:24.576 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-19T01:42:24.577 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-19T01:42:24.577 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-19T01:42:24.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.690+0000 7f16e0379640 10 monclient: get_monmap_and_config 2024-04-19T01:42:24.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.690+0000 7f16e0379640 10 monclient: build_initial_monmap 2024-04-19T01:42:24.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.690+0000 7f16e0379640 10 monclient: monmap: 2024-04-19T01:42:24.693 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:24.693 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:24.693 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:24.697527+0000 2024-04-19T01:42:24.693 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:24.697527+0000 2024-04-19T01:42:24.694 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:24.694 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:24.694 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:24.694 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:24.694 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:24.694 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:24.694 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:24.694 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:24.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16e0379640 1 Processor -- start 2024-04-19T01:42:24.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16e0379640 1 -- start start 2024-04-19T01:42:24.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16e0379640 10 monclient: init 2024-04-19T01:42:24.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16e0379640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:24.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16e0379640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-04-19T01:42:24.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16e0379640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16d8137780 0x7f16d8137b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:24.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16e0379640 10 monclient(hunting): picked mon.noname-b con 0x7f16d8137780 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:24.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16e0379640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f16d81380a0 0x7f16d80a45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:24.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16e0379640 10 monclient(hunting): picked mon.noname-c con 0x7f16d81380a0 addr v2:172.21.15.71:3300/0 2024-04-19T01:42:24.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16e0379640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f16d80a60b0 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:24.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16e0379640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16d809c2e0 con 0x7f16d80a60b0 2024-04-19T01:42:24.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16e0379640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:24.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16e0379640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f16d809bd20 con 0x7f16d81380a0 2024-04-19T01:42:24.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16e0379640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:24.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16e0379640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f16d809c000 con 0x7f16d8137780 2024-04-19T01:42:24.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16e0379640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:24.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16e0379640 10 monclient(hunting): authenticate will time out at 930.517273s 2024-04-19T01:42:24.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16de115640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16d8137780 0x7f16d8137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:24.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16de916640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f16d80a60b0 0x7f16d80a4b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:60998/0 (socket says 172.21.15.62:60998) 2024-04-19T01:42:24.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16de916640 1 -- 172.21.15.62:0/3396530528 learned_addr learned my addr 172.21.15.62:0/3396530528 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:42:24.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16de115640 10 monclient(hunting): get_auth_request con 0x7f16d8137780 auth_method 0 2024-04-19T01:42:24.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16de115640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:24.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16de115640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:24.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16de115640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:24.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16de115640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:24.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16de115640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:24.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16de115640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:24.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16dd914640 1 --2- 172.21.15.62:0/3396530528 >> v2:172.21.15.71:3300/0 conn(0x7f16d81380a0 0x7f16d80a45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:24.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16dd914640 10 monclient(hunting): get_auth_request con 0x7f16d81380a0 auth_method 0 2024-04-19T01:42:24.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16dd914640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:24.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16dd914640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:24.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16dd914640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:24.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16de115640 10 monclient(hunting): handle_auth_done global_id 4208 payload 274 2024-04-19T01:42:24.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16de115640 1 -- 172.21.15.62:0/3396530528 >> v2:172.21.15.71:3300/0 conn(0x7f16d81380a0 msgr2=0x7f16d80a45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:24.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16de115640 1 --2- 172.21.15.62:0/3396530528 >> v2:172.21.15.71:3300/0 conn(0x7f16d81380a0 0x7f16d80a45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:24.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16de115640 1 -- 172.21.15.62:0/3396530528 >> v1:172.21.15.62:6789/0 conn(0x7f16d80a60b0 legacy=0x7f16d80a4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:24.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16de115640 10 monclient: _finish_hunting 0 2024-04-19T01:42:24.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16de115640 1 monclient: found mon.noname-b 2024-04-19T01:42:24.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16de115640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:24.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16de115640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:42:24.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.694+0000 7f16de115640 1 -- 172.21.15.62:0/3396530528 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d8073210 con 0x7f16d8137780 2024-04-19T01:42:24.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.698+0000 7f16de115640 1 -- 172.21.15.62:0/3396530528 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16d8137780 msgr2=0x7f16d8137b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:24.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.698+0000 7f16de115640 1 --2- 172.21.15.62:0/3396530528 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16d8137780 0x7f16d8137b60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f16d40097c0 tx=0x7f16d401c180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4353c6b57255 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:24.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.698+0000 7f16dd113640 1 -- 172.21.15.62:0/3396530528 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d40b0070 con 0x7f16d8137780 2024-04-19T01:42:24.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.698+0000 7f16dd113640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:24.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.698+0000 7f16dd113640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:42:24.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.698+0000 7f16dd113640 10 monclient: dump: 2024-04-19T01:42:24.702 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:24.702 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:24.702 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:24.702 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:24.702 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:24.702 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:24.702 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:24.702 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:24.702 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:24.702 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:24.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.698+0000 7f16dd113640 10 monclient: _finish_auth 0 2024-04-19T01:42:24.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.698+0000 7f16dd113640 10 monclient: _check_auth_tickets 2024-04-19T01:42:24.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.698+0000 7f16dd113640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:24.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.698+0000 7f16dd113640 1 -- 172.21.15.62:0/3396530528 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16d40ab600 con 0x7f16d8137780 2024-04-19T01:42:24.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.698+0000 7f16dd113640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:24.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.698+0000 7f16dd113640 1 -- 172.21.15.62:0/3396530528 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d40ab9d0 con 0x7f16d8137780 2024-04-19T01:42:24.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.698+0000 7f16e0379640 5 monclient: authenticate success, global_id 4208 2024-04-19T01:42:24.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.698+0000 7f16e0379640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:24.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.698+0000 7f16e0379640 4 set_mon_vals no callback set 2024-04-19T01:42:24.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.698+0000 7f16dd113640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:24.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.698+0000 7f16dd113640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:24.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.698+0000 7f16dd113640 10 monclient: dump: 2024-04-19T01:42:24.703 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:24.703 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:24.703 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:24.703 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:24.703 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:24.703 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:24.704 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:24.704 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:24.704 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:24.704 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:24.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.698+0000 7f16e0379640 10 monclient: shutdown 2024-04-19T01:42:24.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.698+0000 7f16e0379640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:24.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.698+0000 7f16e0379640 1 -- 172.21.15.62:0/3396530528 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16d8137780 msgr2=0x7f16d8137b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:24.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.698+0000 7f16e0379640 1 --2- 172.21.15.62:0/3396530528 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16d8137780 0x7f16d8137b60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f16d40097c0 tx=0x7f16d401c180 comp rx=0 tx=0).stop 2024-04-19T01:42:24.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.698+0000 7f16e0379640 1 -- 172.21.15.62:0/3396530528 shutdown_connections 2024-04-19T01:42:24.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.698+0000 7f16e0379640 1 --2- 172.21.15.62:0/3396530528 >> v2:172.21.15.71:3300/0 conn(0x7f16d81380a0 0x7f16d80a45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:24.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.698+0000 7f16e0379640 1 --2- 172.21.15.62:0/3396530528 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16d8137780 0x7f16d8137b60 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:24.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.698+0000 7f16e0379640 1 -- 172.21.15.62:0/3396530528 >> 172.21.15.62:0/3396530528 conn(0x7f16d80aa1d0 msgr2=0x7f16d80a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:24.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.698+0000 7f16e0379640 1 -- 172.21.15.62:0/3396530528 shutdown_connections 2024-04-19T01:42:24.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.698+0000 7f16e0379640 1 -- 172.21.15.62:0/3396530528 wait complete. 2024-04-19T01:42:24.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.698+0000 7f16e0379640 10 monclient: build_initial_monmap 2024-04-19T01:42:24.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.698+0000 7f16e0379640 10 monclient: monmap: 2024-04-19T01:42:24.705 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:24.705 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:24.705 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:24.705 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:24.705 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:24.705 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:24.705 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:24.705 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:24.705 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:24.705 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:24.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.698+0000 7f16e0379640 1 Processor -- start 2024-04-19T01:42:24.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16e0379640 1 -- start start 2024-04-19T01:42:24.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16e0379640 10 monclient: init 2024-04-19T01:42:24.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16e0379640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:24.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16e0379640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-19T01:42:24.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16e0379640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16d80a60b0 0x7f16d81cf400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:24.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16e0379640 10 monclient(hunting): picked mon.noname-c con 0x7f16d80a60b0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:24.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16e0379640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f16d8137780 0x7f16d8136310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:24.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16e0379640 10 monclient(hunting): picked mon.noname-b con 0x7f16d8137780 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:24.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16e0379640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16d81380a0 0x7f16d813c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:24.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16e0379640 10 monclient(hunting): picked mon.noname-a con 0x7f16d81380a0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:24.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16e0379640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:24.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16e0379640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f16d8142170 con 0x7f16d81380a0 2024-04-19T01:42:24.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16e0379640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:24.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16e0379640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f16d80a5430 con 0x7f16d8137780 2024-04-19T01:42:24.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16e0379640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:24.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16e0379640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f16d81422f0 con 0x7f16d80a60b0 2024-04-19T01:42:24.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16e0379640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:24.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16e0379640 10 monclient(hunting): authenticate will time out at 930.524581s 2024-04-19T01:42:24.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16de115640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16d80a60b0 0x7f16d81cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:24.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16de916640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16d81380a0 0x7f16d813c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:24.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16de115640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16d80a60b0 0x7f16d81cf400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:57794/0 (socket says 172.21.15.62:57794) 2024-04-19T01:42:24.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16de115640 1 -- 172.21.15.62:0/2271151933 learned_addr learned my addr 172.21.15.62:0/2271151933 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:24.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16de115640 10 monclient(hunting): get_auth_request con 0x7f16d80a60b0 auth_method 0 2024-04-19T01:42:24.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16de115640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:24.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16de115640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:24.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16de115640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:24.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16de916640 10 monclient(hunting): get_auth_request con 0x7f16d81380a0 auth_method 0 2024-04-19T01:42:24.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16de916640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:24.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16de916640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:24.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16de916640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:24.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16de115640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:24.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16de115640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:24.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16de115640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:24.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16de916640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:24.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16de916640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:24.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16de916640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:24.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16dd914640 1 --2- 172.21.15.62:0/2271151933 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f16d8137780 0x7f16d8136310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:24.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16dd914640 10 monclient(hunting): get_auth_request con 0x7f16d8137780 auth_method 0 2024-04-19T01:42:24.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16dd914640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:24.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16dd914640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:24.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16dd914640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:24.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16de916640 10 monclient(hunting): handle_auth_done global_id 4203 payload 824 2024-04-19T01:42:24.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16de916640 1 -- 172.21.15.62:0/2271151933 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16d80a60b0 msgr2=0x7f16d81cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:24.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16de916640 1 --2- 172.21.15.62:0/2271151933 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16d80a60b0 0x7f16d81cf400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:24.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16de916640 1 -- 172.21.15.62:0/2271151933 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f16d8137780 msgr2=0x7f16d8136310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:24.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16de916640 1 --2- 172.21.15.62:0/2271151933 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f16d8137780 0x7f16d8136310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:24.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16de916640 10 monclient: _finish_hunting 0 2024-04-19T01:42:24.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16de916640 1 monclient: found mon.noname-a 2024-04-19T01:42:24.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16de916640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:24.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16de916640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:24.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16de916640 1 -- 172.21.15.62:0/2271151933 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d80a23e0 con 0x7f16d81380a0 2024-04-19T01:42:24.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16de115640 1 --2- 172.21.15.62:0/2271151933 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16d80a60b0 0x7f16d81cf400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:42:24.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16de916640 1 -- 172.21.15.62:0/2271151933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16d81380a0 msgr2=0x7f16d813c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:24.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16de916640 1 --2- 172.21.15.62:0/2271151933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16d81380a0 0x7f16d813c300 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f16d000caf0 tx=0x7f16d000cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:24.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16ceffd640 1 -- 172.21.15.62:0/2271151933 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d000ad80 con 0x7f16d81380a0 2024-04-19T01:42:24.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:24.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16ceffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:24.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16ceffd640 10 monclient: dump: 2024-04-19T01:42:24.714 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:24.714 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:24.714 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:24.714 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:24.714 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:24.714 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:24.714 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:24.714 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:24.714 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:24.715 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:24.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16ceffd640 10 monclient: _finish_auth 0 2024-04-19T01:42:24.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16ceffd640 10 monclient: _check_auth_tickets 2024-04-19T01:42:24.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:24.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16ceffd640 1 -- 172.21.15.62:0/2271151933 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16d0012400 con 0x7f16d81380a0 2024-04-19T01:42:24.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16ceffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:24.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16ceffd640 1 -- 172.21.15.62:0/2271151933 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d00127f0 con 0x7f16d81380a0 2024-04-19T01:42:24.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:24.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16ceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:24.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16ceffd640 10 monclient: dump: 2024-04-19T01:42:24.715 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:24.715 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:24.715 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:24.716 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:24.716 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:24.716 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:24.716 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:24.716 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:24.716 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:24.716 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:24.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16e0379640 5 monclient: authenticate success, global_id 4203 2024-04-19T01:42:24.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16e0379640 10 monclient: _renew_subs 2024-04-19T01:42:24.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16dc912640 4 set_mon_vals no callback set 2024-04-19T01:42:24.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16e0379640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:24.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16e0379640 1 -- 172.21.15.62:0/2271151933 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16d813ded0 con 0x7f16d81380a0 2024-04-19T01:42:24.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16e0379640 10 monclient: _renew_subs 2024-04-19T01:42:24.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16e0379640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:24.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.702+0000 7f16e0379640 1 -- 172.21.15.62:0/2271151933 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16d81d0280 con 0x7f16d81380a0 2024-04-19T01:42:24.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.706+0000 7f16e0379640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:24.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.706+0000 7f16e0379640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:24.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.706+0000 7f16e0379640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:24.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.706+0000 7f16ceffd640 1 -- 172.21.15.62:0/2271151933 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f16d00129d0 con 0x7f16d81380a0 2024-04-19T01:42:24.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.706+0000 7f16e0379640 1 -- 172.21.15.62:0/2271151933 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16d8068ab0 con 0x7f16d81380a0 2024-04-19T01:42:24.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.706+0000 7f16ceffd640 1 --2- 172.21.15.62:0/2271151933 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f16bc036420 0x7f16bc0388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:24.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.706+0000 7f16ceffd640 1 -- 172.21.15.62:0/2271151933 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f16d00498a0 con 0x7f16d81380a0 2024-04-19T01:42:24.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.706+0000 7f16de115640 1 --2- 172.21.15.62:0/2271151933 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f16bc036420 0x7f16bc0388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:24.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.706+0000 7f16de115640 10 monclient: get_auth_request con 0x7f16bc036420 auth_method 0 2024-04-19T01:42:24.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.710+0000 7f16de115640 1 --2- 172.21.15.62:0/2271151933 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f16bc036420 0x7f16bc0388e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f16d401bfb0 tx=0x7f16d40a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:24.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.710+0000 7f16ceffd640 1 -- 172.21.15.62:0/2271151933 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f16d0048450 con 0x7f16d81380a0 2024-04-19T01:42:24.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.710+0000 7f16ceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:24.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.710+0000 7f16ceffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:24.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.874+0000 7f16e0379640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-19T01:42:24.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.874+0000 7f16e0379640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-19T01:42:24.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.874+0000 7f16e0379640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:24.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.874+0000 7f16e0379640 1 -- 172.21.15.62:0/2271151933 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f16d8068710 con 0x7f16d81380a0 2024-04-19T01:42:24.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.874+0000 7f16ceffd640 1 -- 172.21.15.62:0/2271151933 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+12210 (secure 0 0 0) 0x7f16d001b920 con 0x7f16d81380a0 2024-04-19T01:42:24.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.874+0000 7f16ceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-19T01:42:24.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.874+0000 7f16ceffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:24.876 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:42:24.876 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":15,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","created":"2024-04-19T01:42:09.115311+0000","modified":"2024-04-19T01:42:24.120681+0000","last_up_change":"2024-04-19T01:42:21.102542+0000","last_in_change":"2024-04-19T01:42:12.761455+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-19T01:42:20.748429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-19T01:42:22.720005+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"64304e60-9fe6-413b-8559-3b8ebd179a7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":3870805292},{"type":"v1","addr":"172.21.15.62:6802","nonce":3870805292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":3870805292},{"type":"v1","addr":"172.21.15.62:6806","nonce":3870805292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":3870805292},{"type":"v1","addr":"172.21.15.62:6814","nonce":3870805292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":3870805292},{"type":"v1","addr":"172.21.15.62:6810","nonce":3870805292}]},"public_addr":"172.21.15.62:6802/3870805292","cluster_addr":"172.21.15.62:6806/3870805292","heartbeat_back_addr":"172.21.15.62:6814/3870805292","heartbeat_front_addr":"172.21.15.62:6810/3870805292","state":["exists","up"]},{"osd":1,"uuid":"dec72cc0-fa6d-4fe9-b9cd-19dec591ab68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":199109455},{"type":"v1","addr":"172.21.15.62:6817","nonce":199109455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":199109455},{"type":"v1","addr":"172.21.15.62:6819","nonce":199109455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":199109455},{"type":"v1","addr":"172.21.15.62:6823","nonce":199109455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":199109455},{"type":"v1","addr":"172.21.15.62:6821","nonce":199109455}]},"public_addr":"172.21.15.62:6817/199109455","cluster_addr":"172.21.15.62:6819/199109455","heartbeat_back_addr":"172.21.15.62:6823/199109455","heartbeat_front_addr":"172.21.15.62:6821/199109455","state":["exists","up"]},{"osd":2,"uuid":"2045906a-e0fd-4c41-b73c-27bf7103243b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6801","nonce":1128256055},{"type":"v1","addr":"172.21.15.62:6803","nonce":1128256055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6805","nonce":1128256055},{"type":"v1","addr":"172.21.15.62:6807","nonce":1128256055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6813","nonce":1128256055},{"type":"v1","addr":"172.21.15.62:6815","nonce":1128256055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6809","nonce":1128256055},{"type":"v1","addr":"172.21.15.62:6811","nonce":1128256055}]},"public_addr":"172.21.15.62:6803/1128256055","cluster_addr":"172.21.15.62:6807/1128256055","heartbeat_back_addr":"172.21.15.62:6815/1128256055","heartbeat_front_addr":"172.21.15.62:6811/1128256055","state":["exists","up"]},{"osd":3,"uuid":"230f6e15-548c-4338-a8ec-067b8f85bd2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":2632024688},{"type":"v1","addr":"172.21.15.71:6817","nonce":2632024688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":2632024688},{"type":"v1","addr":"172.21.15.71:6819","nonce":2632024688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":2632024688},{"type":"v1","addr":"172.21.15.71:6823","nonce":2632024688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":2632024688},{"type":"v1","addr":"172.21.15.71:6821","nonce":2632024688}]},"public_addr":"172.21.15.71:6817/2632024688","cluster_addr":"172.21.15.71:6819/2632024688","heartbeat_back_addr":"172.21.15.71:6823/2632024688","heartbeat_front_addr":"172.21.15.71:6821/2632024688","state":["exists","up"]},{"osd":4,"uuid":"06789be2-244f-4b6e-ada2-30b6771676af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":3342848064},{"type":"v1","addr":"172.21.15.71:6809","nonce":3342848064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3342848064},{"type":"v1","addr":"172.21.15.71:6811","nonce":3342848064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":3342848064},{"type":"v1","addr":"172.21.15.71:6815","nonce":3342848064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":3342848064},{"type":"v1","addr":"172.21.15.71:6813","nonce":3342848064}]},"public_addr":"172.21.15.71:6809/3342848064","cluster_addr":"172.21.15.71:6811/3342848064","heartbeat_back_addr":"172.21.15.71:6815/3342848064","heartbeat_front_addr":"172.21.15.71:6813/3342848064","state":["exists","up"]},{"osd":5,"uuid":"552be339-732f-4801-b14d-3d9b08e7f1f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":196330660},{"type":"v1","addr":"172.21.15.71:6801","nonce":196330660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":196330660},{"type":"v1","addr":"172.21.15.71:6803","nonce":196330660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":196330660},{"type":"v1","addr":"172.21.15.71:6807","nonce":196330660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":196330660},{"type":"v1","addr":"172.21.15.71:6805","nonce":196330660}]},"public_addr":"172.21.15.71:6801/196330660","cluster_addr":"172.21.15.71:6803/196330660","heartbeat_back_addr":"172.21.15.71:6807/196330660","heartbeat_front_addr":"172.21.15.71:6805/196330660","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:42:18.255067+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:42:18.295998+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:42:18.304269+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:42:18.274604+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:42:18.338084+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:42:18.149308+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T01:42:24.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.878+0000 7f16e0379640 1 -- 172.21.15.62:0/2271151933 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f16bc036420 msgr2=0x7f16bc0388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:24.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.878+0000 7f16e0379640 1 --2- 172.21.15.62:0/2271151933 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f16bc036420 0x7f16bc0388e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f16d401bfb0 tx=0x7f16d40a6000 comp rx=0 tx=0).stop 2024-04-19T01:42:24.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.878+0000 7f16e0379640 10 monclient: shutdown 2024-04-19T01:42:24.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.878+0000 7f16e0379640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:24.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.878+0000 7f16e0379640 1 -- 172.21.15.62:0/2271151933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16d81380a0 msgr2=0x7f16d813c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:24.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.878+0000 7f16e0379640 1 --2- 172.21.15.62:0/2271151933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16d81380a0 0x7f16d813c300 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f16d000caf0 tx=0x7f16d000cef0 comp rx=0 tx=0).stop 2024-04-19T01:42:24.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.878+0000 7f16e0379640 1 -- 172.21.15.62:0/2271151933 shutdown_connections 2024-04-19T01:42:24.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.878+0000 7f16e0379640 1 --2- 172.21.15.62:0/2271151933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16d81380a0 0x7f16d813c300 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:24.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.878+0000 7f16e0379640 1 --2- 172.21.15.62:0/2271151933 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f16d8137780 0x7f16d8136310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:24.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.878+0000 7f16e0379640 1 --2- 172.21.15.62:0/2271151933 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f16bc036420 0x7f16bc0388e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:24.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.878+0000 7f16e0379640 1 --2- 172.21.15.62:0/2271151933 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f16d80a60b0 0x7f16d81cf400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:24.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.878+0000 7f16e0379640 1 -- 172.21.15.62:0/2271151933 >> 172.21.15.62:0/2271151933 conn(0x7f16d80aa1d0 msgr2=0x7f16d809f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:24.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.878+0000 7f16e0379640 1 -- 172.21.15.62:0/2271151933 shutdown_connections 2024-04-19T01:42:24.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:24.878+0000 7f16e0379640 1 -- 172.21.15.62:0/2271151933 wait complete. 2024-04-19T01:42:24.896 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-19T01:42:24.896 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-19T01:42:25.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.010+0000 7f0e9438f640 10 monclient: get_monmap_and_config 2024-04-19T01:42:25.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.010+0000 7f0e9438f640 10 monclient: build_initial_monmap 2024-04-19T01:42:25.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.010+0000 7f0e9438f640 10 monclient: monmap: 2024-04-19T01:42:25.014 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:25.014 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.014 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:25.016914+0000 2024-04-19T01:42:25.014 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:25.016914+0000 2024-04-19T01:42:25.014 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:25.015 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.015 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:25.015 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:25.015 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:25.015 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:25.015 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:25.016 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.010+0000 7f0e9438f640 1 Processor -- start 2024-04-19T01:42:25.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.010+0000 7f0e9438f640 1 -- start start 2024-04-19T01:42:25.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.010+0000 7f0e9438f640 10 monclient: init 2024-04-19T01:42:25.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9438f640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:25.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9438f640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-04-19T01:42:25.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9438f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e8c0a47e0 0x7f0e8c09f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9438f640 10 monclient(hunting): picked mon.noname-b con 0x7f0e8c0a47e0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:25.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9438f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0e8c09f890 addr v1:172.21.15.71:6789/0 2024-04-19T01:42:25.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9438f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0e8c0a52d0 0x7f0e8c13b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9438f640 10 monclient(hunting): picked mon.noname-a con 0x7f0e8c0a52d0 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:25.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9438f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e8c09ec50 con 0x7f0e8c09f890 2024-04-19T01:42:25.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9438f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9438f640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f0e8c09ef30 con 0x7f0e8c0a52d0 2024-04-19T01:42:25.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9438f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9438f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f0e8c09e690 con 0x7f0e8c0a47e0 2024-04-19T01:42:25.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9438f640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:25.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9438f640 10 monclient(hunting): authenticate will time out at 930.836380s 2024-04-19T01:42:25.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9292c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0e8c0a52d0 0x7f0e8c13b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9212b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e8c0a47e0 0x7f0e8c09f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9212b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e8c0a47e0 0x7f0e8c09f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:57808/0 (socket says 172.21.15.62:57808) 2024-04-19T01:42:25.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9292c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0e8c0a52d0 0x7f0e8c13b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52470/0 (socket says 172.21.15.62:52470) 2024-04-19T01:42:25.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9212b640 1 -- 172.21.15.62:0/1745823911 learned_addr learned my addr 172.21.15.62:0/1745823911 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:25.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9212b640 10 monclient(hunting): get_auth_request con 0x7f0e8c0a47e0 auth_method 0 2024-04-19T01:42:25.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9212b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9212b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9212b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9292c640 10 monclient(hunting): get_auth_request con 0x7f0e8c0a52d0 auth_method 0 2024-04-19T01:42:25.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9292c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9292c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9292c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9292c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9292c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9292c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9212b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9212b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9212b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9292c640 10 monclient(hunting): handle_auth_done global_id 4206 payload 274 2024-04-19T01:42:25.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9292c640 1 -- 172.21.15.62:0/1745823911 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e8c0a47e0 msgr2=0x7f0e8c09f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:25.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9292c640 1 --2- 172.21.15.62:0/1745823911 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e8c0a47e0 0x7f0e8c09f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9292c640 1 -- 172.21.15.62:0/1745823911 >> v1:172.21.15.71:6789/0 conn(0x7f0e8c09f890 legacy=0x7f0e8c0a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:25.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9292c640 10 monclient: _finish_hunting 0 2024-04-19T01:42:25.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9292c640 1 monclient: found mon.noname-a 2024-04-19T01:42:25.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9292c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:25.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9292c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:25.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9292c640 1 -- 172.21.15.62:0/1745823911 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e8c09e970 con 0x7f0e8c0a52d0 2024-04-19T01:42:25.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.014+0000 7f0e9212b640 1 --2- 172.21.15.62:0/1745823911 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e8c0a47e0 0x7f0e8c09f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:42:25.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e9292c640 1 -- 172.21.15.62:0/1745823911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e8c0a52d0 msgr2=0x7f0e8c13b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:25.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e9292c640 1 --2- 172.21.15.62:0/1745823911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e8c0a52d0 0x7f0e8c13b570 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0e88009d90 tx=0x7f0e8801c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=576c0f7e912b9656 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e91129640 1 -- 172.21.15.62:0/1745823911 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e880b2070 con 0x7f0e8c0a52d0 2024-04-19T01:42:25.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e91129640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e91129640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:25.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e91129640 10 monclient: dump: 2024-04-19T01:42:25.035 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.035 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.035 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.035 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.035 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.035 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.035 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.035 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.035 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.035 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e91129640 10 monclient: _finish_auth 0 2024-04-19T01:42:25.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e91129640 10 monclient: _check_auth_tickets 2024-04-19T01:42:25.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e91129640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:25.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e91129640 1 -- 172.21.15.62:0/1745823911 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e880a5920 con 0x7f0e8c0a52d0 2024-04-19T01:42:25.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e91129640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:25.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e91129640 1 -- 172.21.15.62:0/1745823911 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e880a5cf0 con 0x7f0e8c0a52d0 2024-04-19T01:42:25.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e9438f640 5 monclient: authenticate success, global_id 4206 2024-04-19T01:42:25.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e9438f640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:25.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e9438f640 4 set_mon_vals no callback set 2024-04-19T01:42:25.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e91129640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e91129640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:25.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e91129640 10 monclient: dump: 2024-04-19T01:42:25.036 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.036 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.036 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.036 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.036 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.036 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.036 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.036 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.036 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.036 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e9438f640 10 monclient: shutdown 2024-04-19T01:42:25.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e9438f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:25.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e9438f640 1 -- 172.21.15.62:0/1745823911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e8c0a52d0 msgr2=0x7f0e8c13b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e9438f640 1 --2- 172.21.15.62:0/1745823911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e8c0a52d0 0x7f0e8c13b570 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0e88009d90 tx=0x7f0e8801c8c0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e9438f640 1 -- 172.21.15.62:0/1745823911 shutdown_connections 2024-04-19T01:42:25.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e9438f640 1 --2- 172.21.15.62:0/1745823911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e8c0a52d0 0x7f0e8c13b570 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e9438f640 1 --2- 172.21.15.62:0/1745823911 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e8c0a47e0 0x7f0e8c09f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e9438f640 1 -- 172.21.15.62:0/1745823911 >> 172.21.15.62:0/1745823911 conn(0x7f0e8c0a23d0 msgr2=0x7f0e8c0a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:25.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e9438f640 1 -- 172.21.15.62:0/1745823911 shutdown_connections 2024-04-19T01:42:25.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e9438f640 1 -- 172.21.15.62:0/1745823911 wait complete. 2024-04-19T01:42:25.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e9438f640 10 monclient: build_initial_monmap 2024-04-19T01:42:25.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e9438f640 10 monclient: monmap: 2024-04-19T01:42:25.037 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:25.037 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:25.037 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:25.038 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:25.038 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:25.038 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.038 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:25.038 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:25.038 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:25.038 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e9438f640 1 Processor -- start 2024-04-19T01:42:25.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e9438f640 1 -- start start 2024-04-19T01:42:25.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.018+0000 7f0e9438f640 10 monclient: init 2024-04-19T01:42:25.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9438f640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:25.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9438f640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-19T01:42:25.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9438f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e8c09f890 0x7f0e8c1cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9438f640 10 monclient(hunting): picked mon.noname-c con 0x7f0e8c09f890 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:25.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9438f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e8c0a47e0 0x7f0e8c139970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9438f640 10 monclient(hunting): picked mon.noname-a con 0x7f0e8c0a47e0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:25.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9438f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0e8c0a52d0 0x7f0e8c139eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9438f640 10 monclient(hunting): picked mon.noname-b con 0x7f0e8c0a52d0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:25.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9438f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9438f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8c13e800 con 0x7f0e8c0a47e0 2024-04-19T01:42:25.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9438f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9438f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8c13e680 con 0x7f0e8c0a52d0 2024-04-19T01:42:25.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9438f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9438f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f0e8c13e980 con 0x7f0e8c09f890 2024-04-19T01:42:25.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9438f640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:25.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9438f640 10 monclient(hunting): authenticate will time out at 930.844524s 2024-04-19T01:42:25.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9212b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e8c09f890 0x7f0e8c1cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9192a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e8c0a47e0 0x7f0e8c139970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9212b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e8c09f890 0x7f0e8c1cc6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:57810/0 (socket says 172.21.15.62:57810) 2024-04-19T01:42:25.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9212b640 1 -- 172.21.15.62:0/3077061001 learned_addr learned my addr 172.21.15.62:0/3077061001 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:25.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9212b640 10 monclient(hunting): get_auth_request con 0x7f0e8c09f890 auth_method 0 2024-04-19T01:42:25.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9212b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9212b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9212b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9192a640 10 monclient(hunting): get_auth_request con 0x7f0e8c0a47e0 auth_method 0 2024-04-19T01:42:25.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9192a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9192a640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9192a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9212b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9212b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9212b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9192a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9192a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9192a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9292c640 1 --2- 172.21.15.62:0/3077061001 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0e8c0a52d0 0x7f0e8c139eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9212b640 10 monclient(hunting): handle_auth_done global_id 4217 payload 824 2024-04-19T01:42:25.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9212b640 1 -- 172.21.15.62:0/3077061001 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0e8c0a52d0 msgr2=0x7f0e8c139eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9212b640 1 --2- 172.21.15.62:0/3077061001 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0e8c0a52d0 0x7f0e8c139eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9212b640 1 -- 172.21.15.62:0/3077061001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e8c0a47e0 msgr2=0x7f0e8c139970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9212b640 1 --2- 172.21.15.62:0/3077061001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e8c0a47e0 0x7f0e8c139970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9212b640 10 monclient: _finish_hunting 0 2024-04-19T01:42:25.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9212b640 1 monclient: found mon.noname-c 2024-04-19T01:42:25.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9212b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:25.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9212b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9212b640 1 -- 172.21.15.62:0/3077061001 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e8c126a90 con 0x7f0e8c09f890 2024-04-19T01:42:25.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9292c640 10 monclient: get_auth_request con 0x7f0e8c0a52d0 auth_method 0 2024-04-19T01:42:25.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9292c640 1 --2- 172.21.15.62:0/3077061001 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0e8c0a52d0 0x7f0e8c139eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:42:25.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9212b640 1 -- 172.21.15.62:0/3077061001 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e8c09f890 msgr2=0x7f0e8c1cc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:25.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9212b640 1 --2- 172.21.15.62:0/3077061001 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e8c09f890 0x7f0e8c1cc6d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0e8400ea00 tx=0x7f0e8400eec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e82ffd640 1 -- 172.21.15.62:0/3077061001 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e84017070 con 0x7f0e8c09f890 2024-04-19T01:42:25.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e82ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e82ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:25.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e82ffd640 10 monclient: dump: 2024-04-19T01:42:25.042 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.042 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.042 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.042 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.043 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.043 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.043 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.043 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.043 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.043 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e82ffd640 10 monclient: _finish_auth 0 2024-04-19T01:42:25.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e82ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:42:25.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:25.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e82ffd640 1 -- 172.21.15.62:0/3077061001 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e8400cc70 con 0x7f0e8c09f890 2024-04-19T01:42:25.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e82ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:25.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e82ffd640 1 -- 172.21.15.62:0/3077061001 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e84013470 con 0x7f0e8c09f890 2024-04-19T01:42:25.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e82ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e82ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:25.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e82ffd640 10 monclient: dump: 2024-04-19T01:42:25.043 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.044 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.044 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.044 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.044 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.044 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.044 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.044 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.044 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.044 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9438f640 5 monclient: authenticate success, global_id 4217 2024-04-19T01:42:25.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9438f640 10 monclient: _renew_subs 2024-04-19T01:42:25.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9438f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9438f640 1 -- 172.21.15.62:0/3077061001 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e8c125040 con 0x7f0e8c09f890 2024-04-19T01:42:25.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e90928640 4 set_mon_vals no callback set 2024-04-19T01:42:25.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9438f640 10 monclient: _renew_subs 2024-04-19T01:42:25.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9438f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.022+0000 7f0e9438f640 1 -- 172.21.15.62:0/3077061001 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e8c125dc0 con 0x7f0e8c09f890 2024-04-19T01:42:25.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.026+0000 7f0e9438f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:25.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.026+0000 7f0e82ffd640 1 -- 172.21.15.62:0/3077061001 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f0e8401a070 con 0x7f0e8c09f890 2024-04-19T01:42:25.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.026+0000 7f0e9438f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:25.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.026+0000 7f0e9438f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.026+0000 7f0e9438f640 1 -- 172.21.15.62:0/3077061001 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e8c0692b0 con 0x7f0e8c09f890 2024-04-19T01:42:25.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.026+0000 7f0e82ffd640 1 --2- 172.21.15.62:0/3077061001 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0e70036420 0x7f0e700388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.026+0000 7f0e82ffd640 1 -- 172.21.15.62:0/3077061001 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f0e840492d0 con 0x7f0e8c09f890 2024-04-19T01:42:25.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.026+0000 7f0e9192a640 1 --2- 172.21.15.62:0/3077061001 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0e70036420 0x7f0e700388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.026+0000 7f0e9192a640 10 monclient: get_auth_request con 0x7f0e70036420 auth_method 0 2024-04-19T01:42:25.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.026+0000 7f0e9192a640 1 --2- 172.21.15.62:0/3077061001 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0e70036420 0x7f0e700388e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c126980 tx=0x7f0e7c007e40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.030+0000 7f0e82ffd640 1 -- 172.21.15.62:0/3077061001 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0e8c126a90 con 0x7f0e8c09f890 2024-04-19T01:42:25.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.030+0000 7f0e82ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:25.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.030+0000 7f0e82ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:25.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.194+0000 7f0e9438f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-19T01:42:25.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.194+0000 7f0e9438f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-19T01:42:25.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.194+0000 7f0e9438f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.194+0000 7f0e9438f640 1 -- 172.21.15.62:0/3077061001 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0e8c06b980 con 0x7f0e8c09f890 2024-04-19T01:42:25.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.198+0000 7f0e82ffd640 1 -- 172.21.15.62:0/3077061001 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+12210 (secure 0 0 0) 0x7f0e84013e10 con 0x7f0e8c09f890 2024-04-19T01:42:25.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.198+0000 7f0e82ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-19T01:42:25.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.198+0000 7f0e82ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:25.198 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:42:25.199 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":15,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","created":"2024-04-19T01:42:09.115311+0000","modified":"2024-04-19T01:42:24.120681+0000","last_up_change":"2024-04-19T01:42:21.102542+0000","last_in_change":"2024-04-19T01:42:12.761455+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-19T01:42:20.748429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-19T01:42:22.720005+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"64304e60-9fe6-413b-8559-3b8ebd179a7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":3870805292},{"type":"v1","addr":"172.21.15.62:6802","nonce":3870805292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":3870805292},{"type":"v1","addr":"172.21.15.62:6806","nonce":3870805292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":3870805292},{"type":"v1","addr":"172.21.15.62:6814","nonce":3870805292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":3870805292},{"type":"v1","addr":"172.21.15.62:6810","nonce":3870805292}]},"public_addr":"172.21.15.62:6802/3870805292","cluster_addr":"172.21.15.62:6806/3870805292","heartbeat_back_addr":"172.21.15.62:6814/3870805292","heartbeat_front_addr":"172.21.15.62:6810/3870805292","state":["exists","up"]},{"osd":1,"uuid":"dec72cc0-fa6d-4fe9-b9cd-19dec591ab68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":199109455},{"type":"v1","addr":"172.21.15.62:6817","nonce":199109455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":199109455},{"type":"v1","addr":"172.21.15.62:6819","nonce":199109455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":199109455},{"type":"v1","addr":"172.21.15.62:6823","nonce":199109455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":199109455},{"type":"v1","addr":"172.21.15.62:6821","nonce":199109455}]},"public_addr":"172.21.15.62:6817/199109455","cluster_addr":"172.21.15.62:6819/199109455","heartbeat_back_addr":"172.21.15.62:6823/199109455","heartbeat_front_addr":"172.21.15.62:6821/199109455","state":["exists","up"]},{"osd":2,"uuid":"2045906a-e0fd-4c41-b73c-27bf7103243b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6801","nonce":1128256055},{"type":"v1","addr":"172.21.15.62:6803","nonce":1128256055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6805","nonce":1128256055},{"type":"v1","addr":"172.21.15.62:6807","nonce":1128256055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6813","nonce":1128256055},{"type":"v1","addr":"172.21.15.62:6815","nonce":1128256055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6809","nonce":1128256055},{"type":"v1","addr":"172.21.15.62:6811","nonce":1128256055}]},"public_addr":"172.21.15.62:6803/1128256055","cluster_addr":"172.21.15.62:6807/1128256055","heartbeat_back_addr":"172.21.15.62:6815/1128256055","heartbeat_front_addr":"172.21.15.62:6811/1128256055","state":["exists","up"]},{"osd":3,"uuid":"230f6e15-548c-4338-a8ec-067b8f85bd2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":2632024688},{"type":"v1","addr":"172.21.15.71:6817","nonce":2632024688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":2632024688},{"type":"v1","addr":"172.21.15.71:6819","nonce":2632024688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":2632024688},{"type":"v1","addr":"172.21.15.71:6823","nonce":2632024688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":2632024688},{"type":"v1","addr":"172.21.15.71:6821","nonce":2632024688}]},"public_addr":"172.21.15.71:6817/2632024688","cluster_addr":"172.21.15.71:6819/2632024688","heartbeat_back_addr":"172.21.15.71:6823/2632024688","heartbeat_front_addr":"172.21.15.71:6821/2632024688","state":["exists","up"]},{"osd":4,"uuid":"06789be2-244f-4b6e-ada2-30b6771676af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":3342848064},{"type":"v1","addr":"172.21.15.71:6809","nonce":3342848064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3342848064},{"type":"v1","addr":"172.21.15.71:6811","nonce":3342848064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":3342848064},{"type":"v1","addr":"172.21.15.71:6815","nonce":3342848064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":3342848064},{"type":"v1","addr":"172.21.15.71:6813","nonce":3342848064}]},"public_addr":"172.21.15.71:6809/3342848064","cluster_addr":"172.21.15.71:6811/3342848064","heartbeat_back_addr":"172.21.15.71:6815/3342848064","heartbeat_front_addr":"172.21.15.71:6813/3342848064","state":["exists","up"]},{"osd":5,"uuid":"552be339-732f-4801-b14d-3d9b08e7f1f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":196330660},{"type":"v1","addr":"172.21.15.71:6801","nonce":196330660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":196330660},{"type":"v1","addr":"172.21.15.71:6803","nonce":196330660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":196330660},{"type":"v1","addr":"172.21.15.71:6807","nonce":196330660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":196330660},{"type":"v1","addr":"172.21.15.71:6805","nonce":196330660}]},"public_addr":"172.21.15.71:6801/196330660","cluster_addr":"172.21.15.71:6803/196330660","heartbeat_back_addr":"172.21.15.71:6807/196330660","heartbeat_front_addr":"172.21.15.71:6805/196330660","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:42:18.255067+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:42:18.295998+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:42:18.304269+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:42:18.274604+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:42:18.338084+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:42:18.149308+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T01:42:25.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.202+0000 7f0e9438f640 1 -- 172.21.15.62:0/3077061001 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0e70036420 msgr2=0x7f0e700388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.202+0000 7f0e9438f640 1 --2- 172.21.15.62:0/3077061001 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0e70036420 0x7f0e700388e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c126980 tx=0x7f0e7c007e40 comp rx=0 tx=0).stop 2024-04-19T01:42:25.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.202+0000 7f0e9438f640 10 monclient: shutdown 2024-04-19T01:42:25.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.202+0000 7f0e9438f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:25.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.202+0000 7f0e9438f640 1 -- 172.21.15.62:0/3077061001 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e8c09f890 msgr2=0x7f0e8c1cc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.202+0000 7f0e9438f640 1 --2- 172.21.15.62:0/3077061001 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e8c09f890 0x7f0e8c1cc6d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0e8400ea00 tx=0x7f0e8400eec0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.202+0000 7f0e9438f640 1 -- 172.21.15.62:0/3077061001 shutdown_connections 2024-04-19T01:42:25.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.202+0000 7f0e9438f640 1 --2- 172.21.15.62:0/3077061001 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0e8c0a52d0 0x7f0e8c139eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.202+0000 7f0e9438f640 1 --2- 172.21.15.62:0/3077061001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e8c0a47e0 0x7f0e8c139970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.202+0000 7f0e9438f640 1 --2- 172.21.15.62:0/3077061001 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0e70036420 0x7f0e700388e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.202+0000 7f0e9438f640 1 --2- 172.21.15.62:0/3077061001 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0e8c09f890 0x7f0e8c1cc6d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.202+0000 7f0e9438f640 1 -- 172.21.15.62:0/3077061001 >> 172.21.15.62:0/3077061001 conn(0x7f0e8c0a23d0 msgr2=0x7f0e8c13a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:25.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.202+0000 7f0e9438f640 1 -- 172.21.15.62:0/3077061001 shutdown_connections 2024-04-19T01:42:25.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.202+0000 7f0e9438f640 1 -- 172.21.15.62:0/3077061001 wait complete. 2024-04-19T01:42:25.218 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-19T01:42:25.219 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-19T01:42:25.219 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-19T01:42:25.220 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-19T01:42:25.220 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-19T01:42:25.220 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-19T01:42:25.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.334+0000 7f3defa10640 10 monclient: get_monmap_and_config 2024-04-19T01:42:25.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.334+0000 7f3defa10640 10 monclient: build_initial_monmap 2024-04-19T01:42:25.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.334+0000 7f3defa10640 10 monclient: monmap: 2024-04-19T01:42:25.336 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:25.336 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.336 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:25.340248+0000 2024-04-19T01:42:25.336 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:25.340248+0000 2024-04-19T01:42:25.336 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:25.336 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.336 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:25.336 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:25.337 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:25.337 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:25.337 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:25.337 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.334+0000 7f3defa10640 1 Processor -- start 2024-04-19T01:42:25.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.334+0000 7f3defa10640 1 -- start start 2024-04-19T01:42:25.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.334+0000 7f3defa10640 10 monclient: init 2024-04-19T01:42:25.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.334+0000 7f3defa10640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:25.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.334+0000 7f3defa10640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-04-19T01:42:25.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.334+0000 7f3defa10640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3de8137780 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:25.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.334+0000 7f3defa10640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3de80a60b0 0x7f3de813c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.334+0000 7f3defa10640 10 monclient(hunting): picked mon.noname-b con 0x7f3de80a60b0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:25.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.334+0000 7f3defa10640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3de813cc60 addr v1:172.21.15.71:6789/0 2024-04-19T01:42:25.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.334+0000 7f3defa10640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3de809c2e0 con 0x7f3de8137780 2024-04-19T01:42:25.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.334+0000 7f3ded7ac640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f3de8137780 0x7f3de8137b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:33486/0 (socket says 172.21.15.62:33486) 2024-04-19T01:42:25.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.334+0000 7f3decfab640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3de80a60b0 0x7f3de813c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.334+0000 7f3ded7ac640 1 -- 172.21.15.62:0/3181024136 learned_addr learned my addr 172.21.15.62:0/3181024136 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:42:25.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.334+0000 7f3defa10640 1 -- 172.21.15.62:0/3181024136 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3de809c000 con 0x7f3de813cc60 2024-04-19T01:42:25.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.334+0000 7f76dbde5640 10 monclient: get_monmap_and_config 2024-04-19T01:42:25.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.334+0000 7f76dbde5640 10 monclient: build_initial_monmap 2024-04-19T01:42:25.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.334+0000 7f76dbde5640 10 monclient: monmap: 2024-04-19T01:42:25.339 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:25.339 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.339 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:25.341868+0000 2024-04-19T01:42:25.340 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:25.341868+0000 2024-04-19T01:42:25.340 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:25.340 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.340 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:25.340 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:25.340 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:25.340 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:25.340 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:25.340 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.334+0000 7f3defa10640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.334+0000 7f3defa10640 1 -- 172.21.15.62:0/3181024136 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f3de809bd20 con 0x7f3de80a60b0 2024-04-19T01:42:25.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f3defa10640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:25.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f3defa10640 10 monclient(hunting): authenticate will time out at 931.160036s 2024-04-19T01:42:25.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f3decfab640 10 monclient(hunting): get_auth_request con 0x7f3de80a60b0 auth_method 0 2024-04-19T01:42:25.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f3decfab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f3decfab640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f3decfab640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f3decfab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f3decfab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f3decfab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76dbde5640 1 Processor -- start 2024-04-19T01:42:25.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f3decfab640 10 monclient(hunting): handle_auth_done global_id 4220 payload 274 2024-04-19T01:42:25.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f3decfab640 1 -- 172.21.15.62:0/3181024136 >> v1:172.21.15.71:6789/0 conn(0x7f3de813cc60 legacy=0x7f3de813f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:25.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f3decfab640 1 -- 172.21.15.62:0/3181024136 >> v1:172.21.15.62:6789/0 conn(0x7f3de8137780 legacy=0x7f3de8137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:25.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f3decfab640 10 monclient: _finish_hunting 0 2024-04-19T01:42:25.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f3decfab640 1 monclient: found mon.noname-b 2024-04-19T01:42:25.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f3decfab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:25.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f3decfab640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f3decfab640 1 -- 172.21.15.62:0/3181024136 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3de8138270 con 0x7f3de80a60b0 2024-04-19T01:42:25.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76dbde5640 1 -- start start 2024-04-19T01:42:25.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76dbde5640 10 monclient: init 2024-04-19T01:42:25.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76dbde5640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:25.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76dbde5640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-04-19T01:42:25.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76dbde5640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f76d4072560 0x7f76d4072940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76dbde5640 10 monclient(hunting): picked mon.noname-a con 0x7f76d4072560 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:25.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76dbde5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f76d4071a50 addr v1:172.21.15.71:6789/0 2024-04-19T01:42:25.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76dbde5640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f76d4071e30 0x7f76d40a2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76dbde5640 10 monclient(hunting): picked mon.noname-c con 0x7f76d4071e30 addr v2:172.21.15.71:3300/0 2024-04-19T01:42:25.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76dbde5640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76d406f1d0 con 0x7f76d4071a50 2024-04-19T01:42:25.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76dbde5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76dbde5640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f76d40a0fa0 con 0x7f76d4072560 2024-04-19T01:42:25.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76dbde5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76dbde5640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f76d4070680 con 0x7f76d4071e30 2024-04-19T01:42:25.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76dbde5640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:25.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76dbde5640 10 monclient(hunting): authenticate will time out at 931.161079s 2024-04-19T01:42:25.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76d9b81640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f76d4072560 0x7f76d4072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76d9b81640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f76d4072560 0x7f76d4072940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36120/0 (socket says 172.21.15.62:36120) 2024-04-19T01:42:25.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76d9b81640 1 -- 172.21.15.62:0/4270319869 learned_addr learned my addr 172.21.15.62:0/4270319869 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:25.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76d9b81640 10 monclient(hunting): get_auth_request con 0x7f76d4072560 auth_method 0 2024-04-19T01:42:25.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76d9b81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76d9b81640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76d9b81640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76d9b81640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76d9b81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76d9b81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76d9b81640 10 monclient(hunting): handle_auth_done global_id 4212 payload 274 2024-04-19T01:42:25.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76d9b81640 1 -- 172.21.15.62:0/4270319869 >> v2:172.21.15.71:3300/0 conn(0x7f76d4071e30 msgr2=0x7f76d40a2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:25.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76d9b81640 1 --2- 172.21.15.62:0/4270319869 >> v2:172.21.15.71:3300/0 conn(0x7f76d4071e30 0x7f76d40a2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76d9b81640 1 -- 172.21.15.62:0/4270319869 >> v1:172.21.15.71:6789/0 conn(0x7f76d4071a50 legacy=0x7f76d40a2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:25.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76d9b81640 10 monclient: _finish_hunting 0 2024-04-19T01:42:25.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76d9b81640 1 monclient: found mon.noname-a 2024-04-19T01:42:25.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76d9b81640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:25.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76d9b81640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:25.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.338+0000 7f76d9b81640 1 -- 172.21.15.62:0/4270319869 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76d406f3d0 con 0x7f76d4072560 2024-04-19T01:42:25.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3decfab640 1 -- 172.21.15.62:0/3181024136 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3de80a60b0 msgr2=0x7f3de813c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:25.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3decfab640 1 --2- 172.21.15.62:0/3181024136 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3de80a60b0 0x7f3de813c720 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8009820 tx=0x7f3dd801c1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5325f3007452db88 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3ddffff640 1 -- 172.21.15.62:0/3181024136 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd80b0070 con 0x7f3de80a60b0 2024-04-19T01:42:25.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3ddffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3ddffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:42:25.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3ddffff640 10 monclient: dump: 2024-04-19T01:42:25.350 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.350 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.350 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.350 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.350 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.350 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.351 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.351 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.351 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.351 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3ddffff640 10 monclient: _finish_auth 0 2024-04-19T01:42:25.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3ddffff640 10 monclient: _check_auth_tickets 2024-04-19T01:42:25.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3ddffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:25.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3ddffff640 1 -- 172.21.15.62:0/3181024136 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dd80ab600 con 0x7f3de80a60b0 2024-04-19T01:42:25.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3defa10640 5 monclient: authenticate success, global_id 4220 2024-04-19T01:42:25.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3defa10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:42:25.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3ddffff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:25.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3defa10640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:25.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3defa10640 4 set_mon_vals no callback set 2024-04-19T01:42:25.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3ddffff640 1 -- 172.21.15.62:0/3181024136 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd80ab9d0 con 0x7f3de80a60b0 2024-04-19T01:42:25.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3ddffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3ddffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:25.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3ddffff640 10 monclient: dump: 2024-04-19T01:42:25.352 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.352 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.352 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.352 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.352 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.352 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.352 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.352 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.352 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.352 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3defa10640 10 monclient: shutdown 2024-04-19T01:42:25.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3defa10640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:25.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3defa10640 1 -- 172.21.15.62:0/3181024136 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3de80a60b0 msgr2=0x7f3de813c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3defa10640 1 --2- 172.21.15.62:0/3181024136 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3de80a60b0 0x7f3de813c720 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8009820 tx=0x7f3dd801c1a0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3defa10640 1 -- 172.21.15.62:0/3181024136 shutdown_connections 2024-04-19T01:42:25.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3defa10640 1 --2- 172.21.15.62:0/3181024136 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3de80a60b0 0x7f3de813c720 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3defa10640 1 -- 172.21.15.62:0/3181024136 >> 172.21.15.62:0/3181024136 conn(0x7f3de80aa1d0 msgr2=0x7f3de80a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:25.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f76d9b81640 1 -- 172.21.15.62:0/4270319869 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76d4072560 msgr2=0x7f76d4072940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:25.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f76d9b81640 1 --2- 172.21.15.62:0/4270319869 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76d4072560 0x7f76d4072940 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f76d0009870 tx=0x7f76d001c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4043bd8d3d953a46 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3defa10640 1 -- 172.21.15.62:0/3181024136 shutdown_connections 2024-04-19T01:42:25.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3defa10640 1 -- 172.21.15.62:0/3181024136 wait complete. 2024-04-19T01:42:25.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f76d8b7f640 1 -- 172.21.15.62:0/4270319869 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d00b0070 con 0x7f76d4072560 2024-04-19T01:42:25.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f76d8b7f640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f76d8b7f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:25.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f76d8b7f640 10 monclient: dump: 2024-04-19T01:42:25.355 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.355 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.355 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.355 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.355 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.355 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.355 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.356 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.356 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.356 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f76d8b7f640 10 monclient: _finish_auth 0 2024-04-19T01:42:25.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f76d8b7f640 10 monclient: _check_auth_tickets 2024-04-19T01:42:25.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f76d8b7f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:25.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f76dbde5640 5 monclient: authenticate success, global_id 4212 2024-04-19T01:42:25.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f76dbde5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:42:25.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f76d8b7f640 1 -- 172.21.15.62:0/4270319869 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76d00ab600 con 0x7f76d4072560 2024-04-19T01:42:25.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f76d8b7f640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:25.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f76dbde5640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:25.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f76dbde5640 4 set_mon_vals no callback set 2024-04-19T01:42:25.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3defa10640 10 monclient: build_initial_monmap 2024-04-19T01:42:25.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3defa10640 10 monclient: monmap: 2024-04-19T01:42:25.358 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:25.358 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:25.358 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:25.359 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:25.359 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:25.359 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.359 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:25.359 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:25.359 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:25.359 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f76dbde5640 10 monclient: shutdown 2024-04-19T01:42:25.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f76dbde5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:25.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f76dbde5640 1 -- 172.21.15.62:0/4270319869 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76d4072560 msgr2=0x7f76d4072940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f76dbde5640 1 --2- 172.21.15.62:0/4270319869 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76d4072560 0x7f76d4072940 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f76d0009870 tx=0x7f76d001c3b0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f76dbde5640 1 -- 172.21.15.62:0/4270319869 shutdown_connections 2024-04-19T01:42:25.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f76dbde5640 1 --2- 172.21.15.62:0/4270319869 >> v2:172.21.15.71:3300/0 conn(0x7f76d4071e30 0x7f76d40a2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f76dbde5640 1 --2- 172.21.15.62:0/4270319869 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76d4072560 0x7f76d4072940 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f76dbde5640 1 -- 172.21.15.62:0/4270319869 >> 172.21.15.62:0/4270319869 conn(0x7f76d40a6f80 msgr2=0x7f76d40a3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:25.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3defa10640 1 Processor -- start 2024-04-19T01:42:25.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3defa10640 1 -- start start 2024-04-19T01:42:25.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f3defa10640 10 monclient: init 2024-04-19T01:42:25.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f76dbde5640 1 -- 172.21.15.62:0/4270319869 shutdown_connections 2024-04-19T01:42:25.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3defa10640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:25.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3defa10640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-19T01:42:25.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3defa10640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3de80a60b0 0x7f3de81cfa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3defa10640 10 monclient(hunting): picked mon.noname-c con 0x7f3de80a60b0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:25.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3defa10640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3de8137780 0x7f3de81cffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3defa10640 10 monclient(hunting): picked mon.noname-a con 0x7f3de8137780 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:25.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3defa10640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3de813cc60 0x7f3de809e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3ded7ac640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3de80a60b0 0x7f3de81cfa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3ded7ac640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3de80a60b0 0x7f3de81cfa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60268/0 (socket says 172.21.15.62:60268) 2024-04-19T01:42:25.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3decfab640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3de8137780 0x7f3de81cffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f76dbde5640 1 -- 172.21.15.62:0/4270319869 wait complete. 2024-04-19T01:42:25.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f76dbde5640 10 monclient: build_initial_monmap 2024-04-19T01:42:25.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.342+0000 7f76dbde5640 10 monclient: monmap: 2024-04-19T01:42:25.366 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:25.367 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:25.367 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:25.367 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:25.367 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:25.367 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.367 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:25.367 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:25.367 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:25.367 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76dbde5640 1 Processor -- start 2024-04-19T01:42:25.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3decfab640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3de8137780 0x7f3de81cffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36124/0 (socket says 172.21.15.62:36124) 2024-04-19T01:42:25.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3defa10640 10 monclient(hunting): picked mon.noname-b con 0x7f3de813cc60 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:25.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3ded7ac640 1 -- 172.21.15.62:0/2416334283 learned_addr learned my addr 172.21.15.62:0/2416334283 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:25.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3defa10640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3defa10640 1 -- 172.21.15.62:0/2416334283 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3de8143320 con 0x7f3de8137780 2024-04-19T01:42:25.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3defa10640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3defa10640 1 -- 172.21.15.62:0/2416334283 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f3de81431a0 con 0x7f3de813cc60 2024-04-19T01:42:25.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3defa10640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3defa10640 1 -- 172.21.15.62:0/2416334283 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f3de81434a0 con 0x7f3de80a60b0 2024-04-19T01:42:25.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3defa10640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:25.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3dedfad640 1 --2- 172.21.15.62:0/2416334283 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3de813cc60 0x7f3de809e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3defa10640 10 monclient(hunting): authenticate will time out at 931.168961s 2024-04-19T01:42:25.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3ded7ac640 10 monclient(hunting): get_auth_request con 0x7f3de80a60b0 auth_method 0 2024-04-19T01:42:25.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3ded7ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3ded7ac640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3ded7ac640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76dbde5640 1 -- start start 2024-04-19T01:42:25.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76dbde5640 10 monclient: init 2024-04-19T01:42:25.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76dbde5640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:25.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76dbde5640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-19T01:42:25.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76dbde5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f76d4071a50 0x7f76d4075470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76dbde5640 10 monclient(hunting): picked mon.noname-c con 0x7f76d4071a50 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:25.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76dbde5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76d4071e30 0x7f76d4121dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76dbde5640 10 monclient(hunting): picked mon.noname-a con 0x7f76d4071e30 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:25.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76dbde5640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76d4072560 0x7f76d4122310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76dbde5640 10 monclient(hunting): picked mon.noname-b con 0x7f76d4072560 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:25.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76dbde5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76dbde5640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f76d40a3130 con 0x7f76d4071e30 2024-04-19T01:42:25.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76dbde5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76dbde5640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f76d40a2fb0 con 0x7f76d4072560 2024-04-19T01:42:25.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76dbde5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76dbde5640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f76d413ec00 con 0x7f76d4071a50 2024-04-19T01:42:25.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76dbde5640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:25.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76dbde5640 10 monclient(hunting): authenticate will time out at 931.169021s 2024-04-19T01:42:25.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76d9380640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76d4071e30 0x7f76d4121dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76d9380640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76d4071e30 0x7f76d4121dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36140/0 (socket says 172.21.15.62:36140) 2024-04-19T01:42:25.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76d9380640 1 -- 172.21.15.62:0/3430574664 learned_addr learned my addr 172.21.15.62:0/3430574664 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:25.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76d9380640 10 monclient(hunting): get_auth_request con 0x7f76d4071e30 auth_method 0 2024-04-19T01:42:25.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76d9380640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76d9380640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76d9380640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3decfab640 10 monclient(hunting): get_auth_request con 0x7f3de8137780 auth_method 0 2024-04-19T01:42:25.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3decfab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3decfab640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3decfab640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3ded7ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3ded7ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3ded7ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76d9b81640 1 --2- 172.21.15.62:0/3430574664 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f76d4071a50 0x7f76d4075470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76d9b81640 10 monclient(hunting): get_auth_request con 0x7f76d4071a50 auth_method 0 2024-04-19T01:42:25.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76d9b81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76d9b81640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76d9b81640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76da382640 1 --2- 172.21.15.62:0/3430574664 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76d4072560 0x7f76d4122310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3decfab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3decfab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3decfab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76da382640 10 monclient(hunting): get_auth_request con 0x7f76d4072560 auth_method 0 2024-04-19T01:42:25.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76da382640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76da382640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76da382640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3ded7ac640 10 monclient(hunting): handle_auth_done global_id 4223 payload 824 2024-04-19T01:42:25.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3ded7ac640 1 -- 172.21.15.62:0/2416334283 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3de813cc60 msgr2=0x7f3de809e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3ded7ac640 1 --2- 172.21.15.62:0/2416334283 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3de813cc60 0x7f3de809e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3ded7ac640 1 -- 172.21.15.62:0/2416334283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3de8137780 msgr2=0x7f3de81cffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3ded7ac640 1 --2- 172.21.15.62:0/2416334283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3de8137780 0x7f3de81cffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3ded7ac640 10 monclient: _finish_hunting 0 2024-04-19T01:42:25.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3ded7ac640 1 monclient: found mon.noname-c 2024-04-19T01:42:25.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76d9b81640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76d9b81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76d9b81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3ded7ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:25.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3ded7ac640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3ded7ac640 1 -- 172.21.15.62:0/2416334283 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3de80a2030 con 0x7f3de80a60b0 2024-04-19T01:42:25.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76da382640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76da382640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76da382640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76d9b81640 10 monclient(hunting): handle_auth_done global_id 4226 payload 824 2024-04-19T01:42:25.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76d9b81640 1 -- 172.21.15.62:0/3430574664 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76d4072560 msgr2=0x7f76d4122310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76d9b81640 1 --2- 172.21.15.62:0/3430574664 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76d4072560 0x7f76d4122310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76d9b81640 1 -- 172.21.15.62:0/3430574664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76d4071e30 msgr2=0x7f76d4121dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76d9b81640 1 --2- 172.21.15.62:0/3430574664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76d4071e30 0x7f76d4121dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76d9b81640 10 monclient: _finish_hunting 0 2024-04-19T01:42:25.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76d9b81640 1 monclient: found mon.noname-c 2024-04-19T01:42:25.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76d9b81640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:25.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76d9b81640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f76d9b81640 1 -- 172.21.15.62:0/3430574664 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76d40a0330 con 0x7f76d4071a50 2024-04-19T01:42:25.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3dedfad640 10 monclient: get_auth_request con 0x7f3de813cc60 auth_method 0 2024-04-19T01:42:25.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3dedfad640 1 --2- 172.21.15.62:0/2416334283 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3de813cc60 0x7f3de809e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:42:25.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3ded7ac640 1 -- 172.21.15.62:0/2416334283 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3de80a60b0 msgr2=0x7f3de81cfa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:25.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.346+0000 7f3ded7ac640 1 --2- 172.21.15.62:0/2416334283 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3de80a60b0 0x7f3de81cfa60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3de400c470 tx=0x7f3de400c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3dddffb640 1 -- 172.21.15.62:0/2416334283 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de4007070 con 0x7f3de80a60b0 2024-04-19T01:42:25.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3dddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3dddffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:25.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3dddffb640 10 monclient: dump: 2024-04-19T01:42:25.379 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.379 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.379 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.379 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.379 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.379 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.379 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.379 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.379 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.380 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76d9b81640 1 -- 172.21.15.62:0/3430574664 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f76d4071a50 msgr2=0x7f76d4075470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:25.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76d9b81640 1 --2- 172.21.15.62:0/3430574664 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f76d4071a50 0x7f76d4075470 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f76d00ad000 tx=0x7f76d0019150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3dddffb640 10 monclient: _finish_auth 0 2024-04-19T01:42:25.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3dddffb640 10 monclient: _check_auth_tickets 2024-04-19T01:42:25.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3dddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:25.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3dddffb640 1 -- 172.21.15.62:0/2416334283 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3de4007210 con 0x7f3de80a60b0 2024-04-19T01:42:25.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3dddffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:25.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3dddffb640 1 -- 172.21.15.62:0/2416334283 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de4007600 con 0x7f3de80a60b0 2024-04-19T01:42:25.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3dddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3dddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:25.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3dddffb640 10 monclient: dump: 2024-04-19T01:42:25.381 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.381 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.381 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.381 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.381 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.381 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.381 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.381 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.382 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.382 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3ddf7fe640 4 set_mon_vals no callback set 2024-04-19T01:42:25.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3defa10640 5 monclient: authenticate success, global_id 4223 2024-04-19T01:42:25.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76ca7fc640 1 -- 172.21.15.62:0/3430574664 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d00b0050 con 0x7f76d4071a50 2024-04-19T01:42:25.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76ca7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:25.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76ca7fc640 10 monclient: dump: 2024-04-19T01:42:25.383 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.383 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.383 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.383 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.384 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.384 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.384 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.384 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.384 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.384 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76ca7fc640 10 monclient: _finish_auth 0 2024-04-19T01:42:25.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76ca7fc640 10 monclient: _check_auth_tickets 2024-04-19T01:42:25.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:25.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76ca7fc640 1 -- 172.21.15.62:0/3430574664 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76d00ba070 con 0x7f76d4071a50 2024-04-19T01:42:25.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76ca7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:25.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76ca7fc640 1 -- 172.21.15.62:0/3430574664 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d00b5660 con 0x7f76d4071a50 2024-04-19T01:42:25.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76ca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:25.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76ca7fc640 10 monclient: dump: 2024-04-19T01:42:25.385 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.385 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.385 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.385 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.385 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.385 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.385 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.385 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.386 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.386 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3defa10640 10 monclient: _renew_subs 2024-04-19T01:42:25.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3defa10640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3defa10640 1 -- 172.21.15.62:0/2416334283 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3de80a0900 con 0x7f3de80a60b0 2024-04-19T01:42:25.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3defa10640 10 monclient: _renew_subs 2024-04-19T01:42:25.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3defa10640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3defa10640 1 -- 172.21.15.62:0/2416334283 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3de80a8f10 con 0x7f3de80a60b0 2024-04-19T01:42:25.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76dbde5640 5 monclient: authenticate success, global_id 4226 2024-04-19T01:42:25.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76dbde5640 10 monclient: _renew_subs 2024-04-19T01:42:25.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76dbde5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76dbde5640 1 -- 172.21.15.62:0/3430574664 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76d413be50 con 0x7f76d4071a50 2024-04-19T01:42:25.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76dbde5640 10 monclient: _renew_subs 2024-04-19T01:42:25.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76dbde5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76dbde5640 1 -- 172.21.15.62:0/3430574664 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f76d4124c50 con 0x7f76d4071a50 2024-04-19T01:42:25.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76dbde5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76dbde5640 1 -- 172.21.15.62:0/3430574664 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f76d4068ab0 con 0x7f76d4071a50 2024-04-19T01:42:25.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76cbfff640 4 set_mon_vals no callback set 2024-04-19T01:42:25.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3dcb7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3dcb7fe640 1 -- 172.21.15.62:0/2416334283 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3dc4000f90 con 0x7f3de80a60b0 2024-04-19T01:42:25.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76ca7fc640 1 -- 172.21.15.62:0/3430574664 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f76d00ab600 con 0x7f76d4071a50 2024-04-19T01:42:25.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76ca7fc640 1 --2- 172.21.15.62:0/3430574664 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f76b8036470 0x7f76b8038930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76ca7fc640 1 -- 172.21.15.62:0/3430574664 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f76d00e3890 con 0x7f76d4071a50 2024-04-19T01:42:25.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3dddffb640 1 -- 172.21.15.62:0/2416334283 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f3de401b400 con 0x7f3de80a60b0 2024-04-19T01:42:25.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3dddffb640 1 --2- 172.21.15.62:0/2416334283 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f3dcc036470 0x7f3dcc038930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3dddffb640 1 -- 172.21.15.62:0/2416334283 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f3de4048f40 con 0x7f3de80a60b0 2024-04-19T01:42:25.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3dddffb640 1 --2- 172.21.15.62:0/2416334283 >> [v2:172.21.15.62:6800/3870805292,v1:172.21.15.62:6802/3870805292] conn(0x7f3dcc03c360 0x7f3dcc03e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3dddffb640 1 -- 172.21.15.62:0/2416334283 --> [v2:172.21.15.62:6800/3870805292,v1:172.21.15.62:6802/3870805292] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3dcc03ee30 con 0x7f3dcc03c360 2024-04-19T01:42:25.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3dddffb640 1 -- 172.21.15.62:0/2416334283 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7f3de404b510 con 0x7f3de80a60b0 2024-04-19T01:42:25.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3dddffb640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-04-19T01:42:25.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76ca7fc640 1 --2- 172.21.15.62:0/3430574664 >> [v2:172.21.15.71:6808/3342848064,v1:172.21.15.71:6809/3342848064] conn(0x7f76b803c2f0 0x7f76b803e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76ca7fc640 1 -- 172.21.15.62:0/3430574664 --> [v2:172.21.15.71:6808/3342848064,v1:172.21.15.71:6809/3342848064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f76d0007e50 con 0x7f76b803c2f0 2024-04-19T01:42:25.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76ca7fc640 1 -- 172.21.15.62:0/3430574664 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7f76d00ea050 con 0x7f76d4071a50 2024-04-19T01:42:25.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76ca7fc640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-04-19T01:42:25.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3dedfad640 1 --2- 172.21.15.62:0/2416334283 >> [v2:172.21.15.62:6800/3870805292,v1:172.21.15.62:6802/3870805292] conn(0x7f3dcc03c360 0x7f3dcc03e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3decfab640 1 --2- 172.21.15.62:0/2416334283 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f3dcc036470 0x7f3dcc038930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76d9380640 1 --2- 172.21.15.62:0/3430574664 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f76b8036470 0x7f76b8038930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76da382640 1 --2- 172.21.15.62:0/3430574664 >> [v2:172.21.15.71:6808/3342848064,v1:172.21.15.71:6809/3342848064] conn(0x7f76b803c2f0 0x7f76b803e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3dedfad640 10 monclient: get_auth_request con 0x7f3dcc03c360 auth_method 0 2024-04-19T01:42:25.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f3decfab640 10 monclient: get_auth_request con 0x7f3dcc036470 auth_method 0 2024-04-19T01:42:25.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76d9380640 10 monclient: get_auth_request con 0x7f76b8036470 auth_method 0 2024-04-19T01:42:25.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.350+0000 7f76da382640 10 monclient: get_auth_request con 0x7f76b803c2f0 auth_method 0 2024-04-19T01:42:25.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.354+0000 7f3decfab640 1 --2- 172.21.15.62:0/2416334283 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f3dcc036470 0x7f3dcc038930 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3de80a5a70 tx=0x7f3dd80a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.354+0000 7f76d9380640 1 --2- 172.21.15.62:0/3430574664 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f76b8036470 0x7f76b8038930 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f76c4007fd0 tx=0x7f76c4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.354+0000 7f3dedfad640 1 --2- 172.21.15.62:0/2416334283 >> [v2:172.21.15.62:6800/3870805292,v1:172.21.15.62:6802/3870805292] conn(0x7f3dcc03c360 0x7f3dcc03e780 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.354+0000 7f76da382640 1 --2- 172.21.15.62:0/3430574664 >> [v2:172.21.15.71:6808/3342848064,v1:172.21.15.71:6809/3342848064] conn(0x7f76b803c2f0 0x7f76b803e710 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.358+0000 7f3dddffb640 1 -- 172.21.15.62:0/2416334283 <== osd.0 v2:172.21.15.62:6800/3870805292 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3dcc03ee30 con 0x7f3dcc03c360 2024-04-19T01:42:25.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.358+0000 7f76ca7fc640 1 -- 172.21.15.62:0/3430574664 <== osd.4 v2:172.21.15.71:6808/3342848064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f76d0007e50 con 0x7f76b803c2f0 2024-04-19T01:42:25.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.370+0000 7f3dcb7fe640 1 -- 172.21.15.62:0/2416334283 --> [v2:172.21.15.62:6800/3870805292,v1:172.21.15.62:6802/3870805292] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3dc4002c80 con 0x7f3dcc03c360 2024-04-19T01:42:25.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.370+0000 7f3dddffb640 1 -- 172.21.15.62:0/2416334283 <== osd.0 v2:172.21.15.62:6800/3870805292 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3dc4002c80 con 0x7f3dcc03c360 2024-04-19T01:42:25.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.370+0000 7f3dcb7fe640 1 -- 172.21.15.62:0/2416334283 >> [v2:172.21.15.62:6800/3870805292,v1:172.21.15.62:6802/3870805292] conn(0x7f3dcc03c360 msgr2=0x7f3dcc03e780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.370+0000 7f3dcb7fe640 1 --2- 172.21.15.62:0/2416334283 >> [v2:172.21.15.62:6800/3870805292,v1:172.21.15.62:6802/3870805292] conn(0x7f3dcc03c360 0x7f3dcc03e780 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.370+0000 7f3dcb7fe640 1 -- 172.21.15.62:0/2416334283 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f3dcc036470 msgr2=0x7f3dcc038930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.370+0000 7f3dcb7fe640 1 --2- 172.21.15.62:0/2416334283 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f3dcc036470 0x7f3dcc038930 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3de80a5a70 tx=0x7f3dd80a6000 comp rx=0 tx=0).stop 2024-04-19T01:42:25.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.370+0000 7f3dcb7fe640 10 monclient: shutdown 2024-04-19T01:42:25.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.370+0000 7f3dcb7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:25.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.370+0000 7f3dcb7fe640 1 -- 172.21.15.62:0/2416334283 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3de80a60b0 msgr2=0x7f3de81cfa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.370+0000 7f3dcb7fe640 1 --2- 172.21.15.62:0/2416334283 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3de80a60b0 0x7f3de81cfa60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3de400c470 tx=0x7f3de400c930 comp rx=0 tx=0).stop 2024-04-19T01:42:25.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.370+0000 7f3dedfad640 1 -- 172.21.15.62:0/2416334283 reap_dead start 2024-04-19T01:42:25.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.370+0000 7f3dcb7fe640 1 -- 172.21.15.62:0/2416334283 shutdown_connections 2024-04-19T01:42:25.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.370+0000 7f3dcb7fe640 1 -- 172.21.15.62:0/2416334283 >> 172.21.15.62:0/2416334283 conn(0x7f3de80aa1d0 msgr2=0x7f3de80a79d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:25.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.370+0000 7f3dcb7fe640 1 -- 172.21.15.62:0/2416334283 shutdown_connections 2024-04-19T01:42:25.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.370+0000 7f3dcb7fe640 1 -- 172.21.15.62:0/2416334283 wait complete. 2024-04-19T01:42:25.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.374+0000 7f76dbde5640 1 -- 172.21.15.62:0/3430574664 --> [v2:172.21.15.71:6808/3342848064,v1:172.21.15.71:6809/3342848064] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f76d4073180 con 0x7f76b803c2f0 2024-04-19T01:42:25.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.374+0000 7f76ca7fc640 1 -- 172.21.15.62:0/3430574664 <== osd.4 v2:172.21.15.71:6808/3342848064 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f76d4073180 con 0x7f76b803c2f0 2024-04-19T01:42:25.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.374+0000 7f76dbde5640 1 -- 172.21.15.62:0/3430574664 >> [v2:172.21.15.71:6808/3342848064,v1:172.21.15.71:6809/3342848064] conn(0x7f76b803c2f0 msgr2=0x7f76b803e710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.374+0000 7f76dbde5640 1 --2- 172.21.15.62:0/3430574664 >> [v2:172.21.15.71:6808/3342848064,v1:172.21.15.71:6809/3342848064] conn(0x7f76b803c2f0 0x7f76b803e710 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.374+0000 7f76dbde5640 1 -- 172.21.15.62:0/3430574664 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f76b8036470 msgr2=0x7f76b8038930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.374+0000 7f76dbde5640 1 --2- 172.21.15.62:0/3430574664 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f76b8036470 0x7f76b8038930 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f76c4007fd0 tx=0x7f76c4009040 comp rx=0 tx=0).stop 2024-04-19T01:42:25.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.374+0000 7f76dbde5640 10 monclient: shutdown 2024-04-19T01:42:25.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.374+0000 7f76dbde5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:25.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.374+0000 7f76dbde5640 1 -- 172.21.15.62:0/3430574664 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f76d4071a50 msgr2=0x7f76d4075470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.374+0000 7f76dbde5640 1 --2- 172.21.15.62:0/3430574664 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f76d4071a50 0x7f76d4075470 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f76d00ad000 tx=0x7f76d0019150 comp rx=0 tx=0).stop 2024-04-19T01:42:25.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.374+0000 7f76da382640 1 -- 172.21.15.62:0/3430574664 reap_dead start 2024-04-19T01:42:25.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.374+0000 7f76dbde5640 1 -- 172.21.15.62:0/3430574664 shutdown_connections 2024-04-19T01:42:25.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.374+0000 7f76dbde5640 1 -- 172.21.15.62:0/3430574664 >> 172.21.15.62:0/3430574664 conn(0x7f76d40a6f80 msgr2=0x7f76d409e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:25.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.374+0000 7f76dbde5640 1 -- 172.21.15.62:0/3430574664 shutdown_connections 2024-04-19T01:42:25.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.374+0000 7f76dbde5640 1 -- 172.21.15.62:0/3430574664 wait complete. 2024-04-19T01:42:25.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.390+0000 7f380d6c8640 10 monclient: get_monmap_and_config 2024-04-19T01:42:25.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.390+0000 7f380d6c8640 10 monclient: build_initial_monmap 2024-04-19T01:42:25.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.390+0000 7f380d6c8640 10 monclient: monmap: 2024-04-19T01:42:25.399 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:25.399 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.399 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:25.397020+0000 2024-04-19T01:42:25.400 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:25.397020+0000 2024-04-19T01:42:25.400 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:25.400 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.400 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:25.400 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:25.400 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:25.400 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:25.400 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:25.400 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.390+0000 7f380d6c8640 1 Processor -- start 2024-04-19T01:42:25.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.390+0000 7f380d6c8640 1 -- start start 2024-04-19T01:42:25.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.390+0000 7f380d6c8640 10 monclient: init 2024-04-19T01:42:25.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1f191640 10 monclient: get_monmap_and_config 2024-04-19T01:42:25.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1f191640 10 monclient: build_initial_monmap 2024-04-19T01:42:25.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1f191640 10 monclient: monmap: 2024-04-19T01:42:25.401 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:25.401 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.401 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:25.398124+0000 2024-04-19T01:42:25.401 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:25.398124+0000 2024-04-19T01:42:25.401 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:25.402 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.402 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:25.402 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:25.402 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:25.402 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:25.402 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:25.402 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f380d6c8640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:25.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f380d6c8640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-04-19T01:42:25.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f380d6c8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3808137780 addr v1:172.21.15.71:6789/0 2024-04-19T01:42:25.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f380d6c8640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f38080a60b0 0x7f380813c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f380d6c8640 10 monclient(hunting): picked mon.noname-a con 0x7f38080a60b0 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:25.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f380d6c8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f380813cc60 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:25.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f380d6c8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f380809c2e0 con 0x7f380813cc60 2024-04-19T01:42:25.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f38077fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f380813cc60 0x7f380813f050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:33496/0 (socket says 172.21.15.62:33496) 2024-04-19T01:42:25.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f38077fe640 1 -- 172.21.15.62:0/3137093826 learned_addr learned my addr 172.21.15.62:0/3137093826 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:42:25.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f38067fc640 1 --2- 172.21.15.62:0/3137093826 >> v2:172.21.15.62:3300/0 conn(0x7f38080a60b0 0x7f380813c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1f191640 1 Processor -- start 2024-04-19T01:42:25.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f380d6c8640 1 -- 172.21.15.62:0/3137093826 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f380809c000 con 0x7f3808137780 2024-04-19T01:42:25.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f380d6c8640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f380d6c8640 1 -- 172.21.15.62:0/3137093826 --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f380809bd20 con 0x7f38080a60b0 2024-04-19T01:42:25.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f380d6c8640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:25.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f380d6c8640 10 monclient(hunting): authenticate will time out at 931.216827s 2024-04-19T01:42:25.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f38067fc640 10 monclient(hunting): get_auth_request con 0x7f38080a60b0 auth_method 0 2024-04-19T01:42:25.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f38067fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f38067fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f38067fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f38067fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f38067fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f38067fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f38067fc640 10 monclient(hunting): handle_auth_done global_id 4221 payload 274 2024-04-19T01:42:25.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f38067fc640 1 -- 172.21.15.62:0/3137093826 >> v1:172.21.15.71:6789/0 conn(0x7f3808137780 legacy=0x7f3808137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:25.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f38067fc640 1 -- 172.21.15.62:0/3137093826 >> v1:172.21.15.62:6789/0 conn(0x7f380813cc60 legacy=0x7f380813f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:25.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f38067fc640 10 monclient: _finish_hunting 0 2024-04-19T01:42:25.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f38067fc640 1 monclient: found mon.noname-a 2024-04-19T01:42:25.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f38067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:25.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f38067fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:25.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f38067fc640 1 -- 172.21.15.62:0/3137093826 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3808138270 con 0x7f38080a60b0 2024-04-19T01:42:25.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1f191640 1 -- start start 2024-04-19T01:42:25.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1f191640 10 monclient: init 2024-04-19T01:42:25.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1f191640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:25.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1f191640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-04-19T01:42:25.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1f191640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3c180a4ae0 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:25.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1f191640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3c180a45d0 addr v1:172.21.15.71:6789/0 2024-04-19T01:42:25.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1f191640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3c18142200 0x7f3c181445f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1f191640 10 monclient(hunting): picked mon.noname-c con 0x7f3c18142200 addr v2:172.21.15.71:3300/0 2024-04-19T01:42:25.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1f191640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c180a55d0 con 0x7f3c180a4ae0 2024-04-19T01:42:25.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1f191640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c18071800 con 0x7f3c180a45d0 2024-04-19T01:42:25.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1f191640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1f191640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f3c1806dd80 con 0x7f3c18142200 2024-04-19T01:42:25.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1f191640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:25.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1f191640 10 monclient(hunting): authenticate will time out at 931.217662s 2024-04-19T01:42:25.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f40ddc26640 10 monclient: get_monmap_and_config 2024-04-19T01:42:25.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f40ddc26640 10 monclient: build_initial_monmap 2024-04-19T01:42:25.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f40ddc26640 10 monclient: monmap: 2024-04-19T01:42:25.408 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:25.408 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.408 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:25.400665+0000 2024-04-19T01:42:25.408 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:25.400665+0000 2024-04-19T01:42:25.408 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:25.408 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.408 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:25.408 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:25.408 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:25.408 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:25.408 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:25.408 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1cf2d640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f3c180a4ae0 0x7f3c180a4ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:33500/0 (socket says 172.21.15.62:33500) 2024-04-19T01:42:25.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1cf2d640 1 -- 172.21.15.62:0/702340933 learned_addr learned my addr 172.21.15.62:0/702340933 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:42:25.409 INFO:teuthology.orchestra.run.smithi062.stdout:47244640260 2024-04-19T01:42:25.410 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-19T01:42:25.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1d72e640 1 --2- 172.21.15.62:0/702340933 >> v2:172.21.15.71:3300/0 conn(0x7f3c18142200 0x7f3c181445f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1d72e640 10 monclient(hunting): get_auth_request con 0x7f3c18142200 auth_method 0 2024-04-19T01:42:25.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1d72e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1d72e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1d72e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f40ddc26640 1 Processor -- start 2024-04-19T01:42:25.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1d72e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1d72e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1d72e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f40ddc26640 1 -- start start 2024-04-19T01:42:25.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f40ddc26640 10 monclient: init 2024-04-19T01:42:25.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1d72e640 10 monclient(hunting): handle_auth_done global_id 4231 payload 274 2024-04-19T01:42:25.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1d72e640 1 -- 172.21.15.62:0/702340933 >> v1:172.21.15.71:6789/0 conn(0x7f3c180a45d0 legacy=0x7f3c18140ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:25.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1d72e640 1 -- 172.21.15.62:0/702340933 >> v1:172.21.15.62:6789/0 conn(0x7f3c180a4ae0 legacy=0x7f3c180a4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:25.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1d72e640 10 monclient: _finish_hunting 0 2024-04-19T01:42:25.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1d72e640 1 monclient: found mon.noname-c 2024-04-19T01:42:25.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1d72e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:25.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1d72e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-04-19T01:42:25.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f3c1d72e640 1 -- 172.21.15.62:0/702340933 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c1806e060 con 0x7f3c18142200 2024-04-19T01:42:25.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f40ddc26640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:25.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f40ddc26640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-04-19T01:42:25.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f40ddc26640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f40d8137780 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:25.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f40ddc26640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f40d80a60b0 0x7f40d813c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f40ddc26640 10 monclient(hunting): picked mon.noname-a con 0x7f40d80a60b0 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:25.416 INFO:teuthology.orchestra.run.smithi062.stdout:51539607554 2024-04-19T01:42:25.416 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-19T01:42:25.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f40ddc26640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f40d813cc60 addr v1:172.21.15.71:6789/0 2024-04-19T01:42:25.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.394+0000 7f40ddc26640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40d809c2e0 con 0x7f40d8137780 2024-04-19T01:42:25.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f40ddc26640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40d809c000 con 0x7f40d813cc60 2024-04-19T01:42:25.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f40ddc26640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f40ddc26640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f40d809bd20 con 0x7f40d80a60b0 2024-04-19T01:42:25.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f40ddc26640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:25.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f40ddc26640 10 monclient(hunting): authenticate will time out at 931.220026s 2024-04-19T01:42:25.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f40cffff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f40d80a60b0 0x7f40d813c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f40cffff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f40d80a60b0 0x7f40d813c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36166/0 (socket says 172.21.15.62:36166) 2024-04-19T01:42:25.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f40cffff640 1 -- 172.21.15.62:0/380541266 learned_addr learned my addr 172.21.15.62:0/380541266 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:25.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f40cffff640 10 monclient(hunting): get_auth_request con 0x7f40d80a60b0 auth_method 0 2024-04-19T01:42:25.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f40cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f40cffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f40cffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f40cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f40cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f40cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f40cffff640 10 monclient(hunting): handle_auth_done global_id 4224 payload 274 2024-04-19T01:42:25.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f40cffff640 1 -- 172.21.15.62:0/380541266 >> v1:172.21.15.71:6789/0 conn(0x7f40d813cc60 legacy=0x7f40d813f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:25.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f40cffff640 1 -- 172.21.15.62:0/380541266 >> v1:172.21.15.62:6789/0 conn(0x7f40d8137780 legacy=0x7f40d8137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:25.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f40cffff640 10 monclient: _finish_hunting 0 2024-04-19T01:42:25.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f40cffff640 1 monclient: found mon.noname-a 2024-04-19T01:42:25.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f40cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:25.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f40cffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:25.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f40cffff640 1 -- 172.21.15.62:0/380541266 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40d8138270 con 0x7f40d80a60b0 2024-04-19T01:42:25.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f38067fc640 1 -- 172.21.15.62:0/3137093826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38080a60b0 msgr2=0x7f380813c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:25.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f38067fc640 1 --2- 172.21.15.62:0/3137093826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38080a60b0 0x7f380813c720 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f37fc0097c0 tx=0x7f37fc01c210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c73653ed62a9e656 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f3805ffb640 1 -- 172.21.15.62:0/3137093826 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37fc0b0070 con 0x7f38080a60b0 2024-04-19T01:42:25.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f3805ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f3805ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:25.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f3805ffb640 10 monclient: dump: 2024-04-19T01:42:25.422 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.422 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.422 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.422 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.422 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.423 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.423 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.423 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.423 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.423 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f3805ffb640 10 monclient: _finish_auth 0 2024-04-19T01:42:25.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f3805ffb640 10 monclient: _check_auth_tickets 2024-04-19T01:42:25.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f3805ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:25.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f84179c8640 10 monclient: get_monmap_and_config 2024-04-19T01:42:25.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f84179c8640 10 monclient: build_initial_monmap 2024-04-19T01:42:25.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f3805ffb640 1 -- 172.21.15.62:0/3137093826 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37fc0ab600 con 0x7f38080a60b0 2024-04-19T01:42:25.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f380d6c8640 5 monclient: authenticate success, global_id 4221 2024-04-19T01:42:25.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f380d6c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:42:25.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f3805ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:25.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f3805ffb640 1 -- 172.21.15.62:0/3137093826 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37fc0ab9d0 con 0x7f38080a60b0 2024-04-19T01:42:25.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f380d6c8640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:25.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f380d6c8640 4 set_mon_vals no callback set 2024-04-19T01:42:25.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f84179c8640 10 monclient: monmap: 2024-04-19T01:42:25.425 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:25.426 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.426 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:25.404057+0000 2024-04-19T01:42:25.426 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:25.404057+0000 2024-04-19T01:42:25.426 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:25.426 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.426 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:25.426 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:25.426 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:25.427 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:25.427 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:25.427 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f3805ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f3805ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:25.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f3805ffb640 10 monclient: dump: 2024-04-19T01:42:25.427 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.428 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.428 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.428 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.428 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.428 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.428 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.428 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.428 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.428 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f84179c8640 1 Processor -- start 2024-04-19T01:42:25.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f84179c8640 1 -- start start 2024-04-19T01:42:25.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f84179c8640 10 monclient: init 2024-04-19T01:42:25.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f380d6c8640 10 monclient: shutdown 2024-04-19T01:42:25.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f380d6c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:25.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f380d6c8640 1 -- 172.21.15.62:0/3137093826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38080a60b0 msgr2=0x7f380813c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f380d6c8640 1 --2- 172.21.15.62:0/3137093826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38080a60b0 0x7f380813c720 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f37fc0097c0 tx=0x7f37fc01c210 comp rx=0 tx=0).stop 2024-04-19T01:42:25.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f380d6c8640 1 -- 172.21.15.62:0/3137093826 shutdown_connections 2024-04-19T01:42:25.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f380d6c8640 1 --2- 172.21.15.62:0/3137093826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38080a60b0 0x7f380813c720 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f37fc0097c0 tx=0x7f37fc01c210 comp rx=0 tx=0).stop 2024-04-19T01:42:25.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f380d6c8640 1 -- 172.21.15.62:0/3137093826 >> 172.21.15.62:0/3137093826 conn(0x7f38080aa1d0 msgr2=0x7f38080a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:25.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f380d6c8640 1 -- 172.21.15.62:0/3137093826 shutdown_connections 2024-04-19T01:42:25.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f380d6c8640 1 -- 172.21.15.62:0/3137093826 wait complete. 2024-04-19T01:42:25.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f84179c8640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:25.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f84179c8640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-04-19T01:42:25.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f84179c8640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f84100a2360 0x7f8410072bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f84179c8640 10 monclient(hunting): picked mon.noname-a con 0x7f84100a2360 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:25.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f84179c8640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f84100a2740 0x7f84100a2b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f84179c8640 10 monclient(hunting): picked mon.noname-c con 0x7f84100a2740 addr v2:172.21.15.71:3300/0 2024-04-19T01:42:25.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f84179c8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f84101351b0 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:25.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f84179c8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8410075e00 con 0x7f84101351b0 2024-04-19T01:42:25.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f84179c8640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f84179c8640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f8410072260 con 0x7f84100a2360 2024-04-19T01:42:25.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f84179c8640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f84179c8640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f84100730f0 con 0x7f84100a2740 2024-04-19T01:42:25.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f84179c8640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:25.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f84179c8640 10 monclient(hunting): authenticate will time out at 931.223218s 2024-04-19T01:42:25.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f380d6c8640 10 monclient: build_initial_monmap 2024-04-19T01:42:25.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f380d6c8640 10 monclient: monmap: 2024-04-19T01:42:25.433 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:25.433 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:25.433 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:25.433 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:25.433 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:25.433 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.433 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:25.433 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:25.433 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:25.433 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f8414f63640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f84100a2740 0x7f84100a2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f8414f63640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f84100a2740 0x7f84100a2b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.62:41176/0 (socket says 172.21.15.62:41176) 2024-04-19T01:42:25.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f8414f63640 1 -- 172.21.15.62:0/2332247771 learned_addr learned my addr 172.21.15.62:0/2332247771 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:25.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f8414f63640 10 monclient(hunting): get_auth_request con 0x7f84100a2740 auth_method 0 2024-04-19T01:42:25.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f8414f63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f8414f63640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f8414f63640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f8414f63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f8414f63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f8414f63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f380d6c8640 1 Processor -- start 2024-04-19T01:42:25.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f380d6c8640 1 -- start start 2024-04-19T01:42:25.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.398+0000 7f380d6c8640 10 monclient: init 2024-04-19T01:42:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f8414f63640 10 monclient(hunting): handle_auth_done global_id 4234 payload 274 2024-04-19T01:42:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f8414f63640 1 -- 172.21.15.62:0/2332247771 >> v2:172.21.15.62:3300/0 conn(0x7f84100a2360 msgr2=0x7f8410072bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f8414f63640 1 --2- 172.21.15.62:0/2332247771 >> v2:172.21.15.62:3300/0 conn(0x7f84100a2360 0x7f8410072bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f8414f63640 1 -- 172.21.15.62:0/2332247771 >> v1:172.21.15.62:6789/0 conn(0x7f84101351b0 legacy=0x7f8410135620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f8414f63640 10 monclient: _finish_hunting 0 2024-04-19T01:42:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f8414f63640 1 monclient: found mon.noname-c 2024-04-19T01:42:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f8414f63640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:25.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f8414f63640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-04-19T01:42:25.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f8414f63640 1 -- 172.21.15.62:0/2332247771 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84100759c0 con 0x7f84100a2740 2024-04-19T01:42:25.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f380d6c8640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:25.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f380d6c8640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-19T01:42:25.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f380d6c8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3808137780 0x7f38081cf610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f380d6c8640 10 monclient(hunting): picked mon.noname-a con 0x7f3808137780 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:25.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f380d6c8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f380813cc60 0x7f3808136480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f380d6c8640 10 monclient(hunting): picked mon.noname-b con 0x7f380813cc60 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:25.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f380d6c8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f380809e840 0x7f380809ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f380d6c8640 10 monclient(hunting): picked mon.noname-c con 0x7f380809e840 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:25.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f380d6c8640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f380d6c8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3808143320 con 0x7f3808137780 2024-04-19T01:42:25.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f380d6c8640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f380d6c8640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f38081431a0 con 0x7f380813cc60 2024-04-19T01:42:25.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f380d6c8640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f380d6c8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f38081434a0 con 0x7f380809e840 2024-04-19T01:42:25.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f380d6c8640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:25.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f380d6c8640 10 monclient(hunting): authenticate will time out at 931.224502s 2024-04-19T01:42:25.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f40cffff640 1 -- 172.21.15.62:0/380541266 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40d80a60b0 msgr2=0x7f40d813c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:25.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3806ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3808137780 0x7f38081cf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f40cffff640 1 --2- 172.21.15.62:0/380541266 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40d80a60b0 0x7f40d813c720 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f40c0009870 tx=0x7f40c001c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efb548408e63ac29 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3806ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3808137780 0x7f38081cf610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36176/0 (socket says 172.21.15.62:36176) 2024-04-19T01:42:25.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3806ffd640 1 -- 172.21.15.62:0/1898297249 learned_addr learned my addr 172.21.15.62:0/1898297249 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:25.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3806ffd640 10 monclient(hunting): get_auth_request con 0x7f3808137780 auth_method 0 2024-04-19T01:42:25.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3806ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3806ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3806ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f38077fe640 1 --2- 172.21.15.62:0/1898297249 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f380809e840 0x7f380809ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f38077fe640 10 monclient(hunting): get_auth_request con 0x7f380809e840 auth_method 0 2024-04-19T01:42:25.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f38077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f38077fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f38077fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3806ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3806ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3806ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f38077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f38077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f38077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3806ffd640 10 monclient(hunting): handle_auth_done global_id 4227 payload 824 2024-04-19T01:42:25.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3806ffd640 1 -- 172.21.15.62:0/1898297249 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f380809e840 msgr2=0x7f380809ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3806ffd640 1 --2- 172.21.15.62:0/1898297249 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f380809e840 0x7f380809ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f38067fc640 1 --2- 172.21.15.62:0/1898297249 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f380813cc60 0x7f3808136480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3806ffd640 1 -- 172.21.15.62:0/1898297249 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f380813cc60 msgr2=0x7f3808136480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3806ffd640 1 --2- 172.21.15.62:0/1898297249 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f380813cc60 0x7f3808136480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3806ffd640 10 monclient: _finish_hunting 0 2024-04-19T01:42:25.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3806ffd640 1 monclient: found mon.noname-a 2024-04-19T01:42:25.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3806ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:25.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3806ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:25.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3806ffd640 1 -- 172.21.15.62:0/1898297249 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38080a1f30 con 0x7f3808137780 2024-04-19T01:42:25.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f38067fc640 10 monclient: get_auth_request con 0x7f380813cc60 auth_method 0 2024-04-19T01:42:25.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f38067fc640 1 --2- 172.21.15.62:0/1898297249 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f380813cc60 0x7f3808136480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:42:25.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1d72e640 1 -- 172.21.15.62:0/702340933 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c18142200 msgr2=0x7f3c181445f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:25.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1d72e640 1 --2- 172.21.15.62:0/702340933 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c18142200 0x7f3c181445f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3c10009d90 tx=0x7f3c1001c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5cb3489e09fd3d5b server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c177fe640 1 -- 172.21.15.62:0/702340933 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c100b2070 con 0x7f3c18142200 2024-04-19T01:42:25.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c177fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:25.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c177fe640 10 monclient: dump: 2024-04-19T01:42:25.444 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.444 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.444 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.444 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.444 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.444 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.444 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.444 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.444 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.445 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c177fe640 10 monclient: _finish_auth 0 2024-04-19T01:42:25.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c177fe640 10 monclient: _check_auth_tickets 2024-04-19T01:42:25.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:25.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 5 monclient: authenticate success, global_id 4231 2024-04-19T01:42:25.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:42:25.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c177fe640 1 -- 172.21.15.62:0/702340933 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c100a5860 con 0x7f3c18142200 2024-04-19T01:42:25.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c177fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:25.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c177fe640 1 -- 172.21.15.62:0/702340933 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c100a5c30 con 0x7f3c18142200 2024-04-19T01:42:25.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:42:25.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c177fe640 10 monclient: dump: 2024-04-19T01:42:25.445 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.445 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.445 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.446 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.446 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.446 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.446 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.446 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.446 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.446 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:25.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 4 set_mon_vals no callback set 2024-04-19T01:42:25.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 10 monclient: shutdown 2024-04-19T01:42:25.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:25.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 1 -- 172.21.15.62:0/702340933 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c18142200 msgr2=0x7f3c181445f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 1 --2- 172.21.15.62:0/702340933 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c18142200 0x7f3c181445f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3c10009d90 tx=0x7f3c1001c830 comp rx=0 tx=0).stop 2024-04-19T01:42:25.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 1 -- 172.21.15.62:0/702340933 shutdown_connections 2024-04-19T01:42:25.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 1 --2- 172.21.15.62:0/702340933 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c18142200 0x7f3c181445f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 1 -- 172.21.15.62:0/702340933 >> 172.21.15.62:0/702340933 conn(0x7f3c180aa1d0 msgr2=0x7f3c180a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:25.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 1 -- 172.21.15.62:0/702340933 shutdown_connections 2024-04-19T01:42:25.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 1 -- 172.21.15.62:0/702340933 wait complete. 2024-04-19T01:42:25.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 10 monclient: build_initial_monmap 2024-04-19T01:42:25.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 10 monclient: monmap: 2024-04-19T01:42:25.447 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:25.447 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:25.447 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:25.447 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:25.447 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:25.447 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.447 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:25.447 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:25.447 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:25.447 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f40d6ffd640 1 -- 172.21.15.62:0/380541266 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c00b0070 con 0x7f40d80a60b0 2024-04-19T01:42:25.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f40d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f40d6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:25.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f40d6ffd640 10 monclient: dump: 2024-04-19T01:42:25.448 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.448 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.448 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.449 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.449 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.449 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.449 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.449 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.449 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.449 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f40d6ffd640 10 monclient: _finish_auth 0 2024-04-19T01:42:25.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f40d6ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:42:25.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f40d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:25.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f40d6ffd640 1 -- 172.21.15.62:0/380541266 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40c00ab600 con 0x7f40d80a60b0 2024-04-19T01:42:25.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f40ddc26640 5 monclient: authenticate success, global_id 4224 2024-04-19T01:42:25.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f40ddc26640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:42:25.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f40d6ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:25.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f40ddc26640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:25.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f40ddc26640 4 set_mon_vals no callback set 2024-04-19T01:42:25.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f40d6ffd640 1 -- 172.21.15.62:0/380541266 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c00ab9d0 con 0x7f40d80a60b0 2024-04-19T01:42:25.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f40d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f40d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:25.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f40d6ffd640 10 monclient: dump: 2024-04-19T01:42:25.450 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.450 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.450 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.450 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.450 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.450 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.450 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.450 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.451 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.451 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 1 Processor -- start 2024-04-19T01:42:25.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 1 -- start start 2024-04-19T01:42:25.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 10 monclient: init 2024-04-19T01:42:25.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:25.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-19T01:42:25.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c180a45d0 0x7f3c1813ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 10 monclient(hunting): picked mon.noname-b con 0x7f3c180a45d0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:25.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c180a4ae0 0x7f3c1813b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 10 monclient(hunting): picked mon.noname-a con 0x7f3c180a4ae0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:25.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3c18142200 0x7f3c18140d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 10 monclient(hunting): picked mon.noname-c con 0x7f3c18142200 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:25.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3c18147970 con 0x7f3c180a4ae0 2024-04-19T01:42:25.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f3c180a63e0 con 0x7f3c180a45d0 2024-04-19T01:42:25.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f3c18147af0 con 0x7f3c18142200 2024-04-19T01:42:25.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:25.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1f191640 10 monclient(hunting): authenticate will time out at 931.227758s 2024-04-19T01:42:25.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3806ffd640 1 -- 172.21.15.62:0/1898297249 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3808137780 msgr2=0x7f38081cf610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:25.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3806ffd640 1 --2- 172.21.15.62:0/1898297249 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3808137780 0x7f38081cf610 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f37f800c480 tx=0x7f37f800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f37ef7fe640 1 -- 172.21.15.62:0/1898297249 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f8012a40 con 0x7f3808137780 2024-04-19T01:42:25.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f37ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f37ef7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:25.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f37ef7fe640 10 monclient: dump: 2024-04-19T01:42:25.454 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.454 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.454 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.454 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.454 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.454 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.454 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.454 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.454 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.454 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f37ef7fe640 10 monclient: _finish_auth 0 2024-04-19T01:42:25.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f37ef7fe640 10 monclient: _check_auth_tickets 2024-04-19T01:42:25.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f37ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:25.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f380d6c8640 5 monclient: authenticate success, global_id 4227 2024-04-19T01:42:25.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f380d6c8640 10 monclient: _renew_subs 2024-04-19T01:42:25.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f380d6c8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:25.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f380d6c8640 1 -- 172.21.15.62:0/1898297249 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38081d0500 con 0x7f3808137780 2024-04-19T01:42:25.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f380d6c8640 10 monclient: _renew_subs 2024-04-19T01:42:25.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f380d6c8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:25.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f380d6c8640 1 -- 172.21.15.62:0/1898297249 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38080a04e0 con 0x7f3808137780 2024-04-19T01:42:25.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f37ef7fe640 1 -- 172.21.15.62:0/1898297249 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37f8017070 con 0x7f3808137780 2024-04-19T01:42:25.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f37ef7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:25.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f37ef7fe640 1 -- 172.21.15.62:0/1898297249 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f8012e40 con 0x7f3808137780 2024-04-19T01:42:25.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f37ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f37ef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:25.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f38057fa640 4 set_mon_vals no callback set 2024-04-19T01:42:25.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f37ef7fe640 10 monclient: dump: 2024-04-19T01:42:25.455 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.455 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.455 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.456 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.456 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.456 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.456 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.456 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.456 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.456 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1d72e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3c18142200 0x7f3c18140d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1d72e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3c18142200 0x7f3c18140d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60292/0 (socket says 172.21.15.62:60292) 2024-04-19T01:42:25.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1d72e640 1 -- 172.21.15.62:0/2463928378 learned_addr learned my addr 172.21.15.62:0/2463928378 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:25.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1d72e640 10 monclient(hunting): get_auth_request con 0x7f3c18142200 auth_method 0 2024-04-19T01:42:25.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1d72e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1d72e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f3c1d72e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f40ddc26640 10 monclient: shutdown 2024-04-19T01:42:25.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f40ddc26640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:25.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f40ddc26640 1 -- 172.21.15.62:0/380541266 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40d80a60b0 msgr2=0x7f40d813c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.402+0000 7f40ddc26640 1 --2- 172.21.15.62:0/380541266 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40d80a60b0 0x7f40d813c720 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f40c0009870 tx=0x7f40c001c2d0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c1d72e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c1d72e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c1d72e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c1d72e640 10 monclient(hunting): handle_auth_done global_id 4232 payload 824 2024-04-19T01:42:25.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c1d72e640 1 -- 172.21.15.62:0/2463928378 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c180a45d0 msgr2=0x7f3c1813ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c1d72e640 1 --2- 172.21.15.62:0/2463928378 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c180a45d0 0x7f3c1813ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c1d72e640 1 -- 172.21.15.62:0/2463928378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c180a4ae0 msgr2=0x7f3c1813b040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:42:25.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c1d72e640 1 --2- 172.21.15.62:0/2463928378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c180a4ae0 0x7f3c1813b040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c1d72e640 10 monclient: _finish_hunting 0 2024-04-19T01:42:25.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c1d72e640 1 monclient: found mon.noname-c 2024-04-19T01:42:25.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c1d72e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:25.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c1d72e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c1d72e640 1 -- 172.21.15.62:0/2463928378 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c1813e830 con 0x7f3c18142200 2024-04-19T01:42:25.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f380d6c8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:25.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f380d6c8640 1 -- 172.21.15.62:0/1898297249 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f37e8000f90 con 0x7f3808137780 2024-04-19T01:42:25.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f37ef7fe640 1 -- 172.21.15.62:0/1898297249 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f37f801c460 con 0x7f3808137780 2024-04-19T01:42:25.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f37ef7fe640 1 --2- 172.21.15.62:0/1898297249 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f37f4036420 0x7f37f40388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f37ef7fe640 1 -- 172.21.15.62:0/1898297249 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f37f8048fd0 con 0x7f3808137780 2024-04-19T01:42:25.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40ddc26640 1 -- 172.21.15.62:0/380541266 shutdown_connections 2024-04-19T01:42:25.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40ddc26640 1 --2- 172.21.15.62:0/380541266 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40d80a60b0 0x7f40d813c720 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40ddc26640 1 -- 172.21.15.62:0/380541266 >> 172.21.15.62:0/380541266 conn(0x7f40d80aa1d0 msgr2=0x7f40d80a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:25.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f37ef7fe640 1 --2- 172.21.15.62:0/1898297249 >> [v2:172.21.15.71:6800/196330660,v1:172.21.15.71:6801/196330660] conn(0x7f37f403c290 0x7f37f403e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f37ef7fe640 1 -- 172.21.15.62:0/1898297249 --> [v2:172.21.15.71:6800/196330660,v1:172.21.15.71:6801/196330660] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f37f403ed60 con 0x7f37f403c290 2024-04-19T01:42:25.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f37ef7fe640 1 -- 172.21.15.62:0/1898297249 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7f37f804b5a0 con 0x7f3808137780 2024-04-19T01:42:25.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f37ef7fe640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-04-19T01:42:25.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40ddc26640 1 -- 172.21.15.62:0/380541266 shutdown_connections 2024-04-19T01:42:25.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40ddc26640 1 -- 172.21.15.62:0/380541266 wait complete. 2024-04-19T01:42:25.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f38067fc640 1 --2- 172.21.15.62:0/1898297249 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f37f4036420 0x7f37f40388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f38067fc640 10 monclient: get_auth_request con 0x7f37f4036420 auth_method 0 2024-04-19T01:42:25.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f38077fe640 1 --2- 172.21.15.62:0/1898297249 >> [v2:172.21.15.71:6800/196330660,v1:172.21.15.71:6801/196330660] conn(0x7f37f403c290 0x7f37f403e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f38077fe640 10 monclient: get_auth_request con 0x7f37f403c290 auth_method 0 2024-04-19T01:42:25.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f8414f63640 1 -- 172.21.15.62:0/2332247771 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84100a2740 msgr2=0x7f84100a2b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:25.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f8414f63640 1 --2- 172.21.15.62:0/2332247771 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84100a2740 0x7f84100a2b40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f841006ea70 tx=0x7f840001c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=814fbf79300a9429 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40ddc26640 10 monclient: build_initial_monmap 2024-04-19T01:42:25.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40ddc26640 10 monclient: monmap: 2024-04-19T01:42:25.463 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:25.463 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:25.463 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:25.463 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:25.463 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:25.463 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.463 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:25.463 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:25.463 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:25.464 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f8407fff640 1 -- 172.21.15.62:0/2332247771 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84000b1070 con 0x7f84100a2740 2024-04-19T01:42:25.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f8407fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f8407fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:25.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f8407fff640 10 monclient: dump: 2024-04-19T01:42:25.464 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.464 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.465 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.465 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.465 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.465 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.465 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.465 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.465 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.465 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f8407fff640 10 monclient: _finish_auth 0 2024-04-19T01:42:25.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f8407fff640 10 monclient: _check_auth_tickets 2024-04-19T01:42:25.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f8407fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:25.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f8407fff640 1 -- 172.21.15.62:0/2332247771 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84000ac620 con 0x7f84100a2740 2024-04-19T01:42:25.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f8407fff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:25.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f8407fff640 1 -- 172.21.15.62:0/2332247771 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84000ac9f0 con 0x7f84100a2740 2024-04-19T01:42:25.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f8407fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f8407fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:42:25.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f8407fff640 10 monclient: dump: 2024-04-19T01:42:25.466 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.466 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.466 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.466 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.466 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.466 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.466 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.466 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.466 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.466 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f84179c8640 5 monclient: authenticate success, global_id 4234 2024-04-19T01:42:25.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f84179c8640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:25.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f84179c8640 4 set_mon_vals no callback set 2024-04-19T01:42:25.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40ddc26640 1 Processor -- start 2024-04-19T01:42:25.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40ddc26640 1 -- start start 2024-04-19T01:42:25.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40ddc26640 10 monclient: init 2024-04-19T01:42:25.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c1d72e640 1 -- 172.21.15.62:0/2463928378 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3c18142200 msgr2=0x7f3c18140d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:25.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c1d72e640 1 --2- 172.21.15.62:0/2463928378 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3c18142200 0x7f3c18140d70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3c100b0040 tx=0x7f3c100195d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40ddc26640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:25.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40ddc26640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-19T01:42:25.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40ddc26640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f40d80a60b0 0x7f40d81cf720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40ddc26640 10 monclient(hunting): picked mon.noname-b con 0x7f40d80a60b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:25.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40ddc26640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40d8137780 0x7f40d81cfef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40ddc26640 10 monclient(hunting): picked mon.noname-a con 0x7f40d8137780 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:25.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40ddc26640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40d813cc60 0x7f40d809e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40ddc26640 10 monclient(hunting): picked mon.noname-c con 0x7f40d813cc60 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:25.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40ddc26640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40ddc26640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f40d8143320 con 0x7f40d8137780 2024-04-19T01:42:25.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40ddc26640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40ddc26640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f40d81431a0 con 0x7f40d80a60b0 2024-04-19T01:42:25.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40ddc26640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40ddc26640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f40d81434a0 con 0x7f40d813cc60 2024-04-19T01:42:25.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40ddc26640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:25.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40ddc26640 10 monclient(hunting): authenticate will time out at 931.230849s 2024-04-19T01:42:25.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c1d72e640 1 --2- 172.21.15.62:0/2463928378 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3c18142200 0x7f3c18140d70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3c100b0040 tx=0x7f3c100195d0 comp rx=0 tx=0).queue_received will delay after 1.71349e+09 on 0x7f3c100b8880 config(0 keys) 2024-04-19T01:42:25.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c157fa640 1 -- 172.21.15.62:0/2463928378 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c100bd020 con 0x7f3c18142200 2024-04-19T01:42:25.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f84179c8640 10 monclient: shutdown 2024-04-19T01:42:25.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f84179c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:25.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f84179c8640 1 -- 172.21.15.62:0/2332247771 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84100a2740 msgr2=0x7f84100a2b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f84179c8640 1 --2- 172.21.15.62:0/2332247771 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84100a2740 0x7f84100a2b40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f841006ea70 tx=0x7f840001c3e0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c157fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:25.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c157fa640 10 monclient: dump: 2024-04-19T01:42:25.470 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.470 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.470 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.470 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.470 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.470 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.470 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.471 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.471 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.471 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c157fa640 10 monclient: _finish_auth 0 2024-04-19T01:42:25.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c157fa640 10 monclient: _check_auth_tickets 2024-04-19T01:42:25.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:25.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c157fa640 1 -- 172.21.15.62:0/2463928378 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c100b8880 con 0x7f3c18142200 2024-04-19T01:42:25.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c157fa640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:25.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c1f191640 5 monclient: authenticate success, global_id 4232 2024-04-19T01:42:25.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c1f191640 10 monclient: _renew_subs 2024-04-19T01:42:25.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f84179c8640 1 -- 172.21.15.62:0/2332247771 shutdown_connections 2024-04-19T01:42:25.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f84179c8640 1 --2- 172.21.15.62:0/2332247771 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84100a2740 0x7f84100a2b40 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f84179c8640 1 --2- 172.21.15.62:0/2332247771 >> v2:172.21.15.62:3300/0 conn(0x7f84100a2360 0x7f8410072bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f84179c8640 1 -- 172.21.15.62:0/2332247771 >> 172.21.15.62:0/2332247771 conn(0x7f84100a7070 msgr2=0x7f84100a3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:25.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c1f191640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c1f191640 1 -- 172.21.15.62:0/2463928378 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c1813b580 con 0x7f3c18142200 2024-04-19T01:42:25.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40d7fff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40d813cc60 0x7f40d809e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f84179c8640 1 -- 172.21.15.62:0/2332247771 shutdown_connections 2024-04-19T01:42:25.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40d7fff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40d813cc60 0x7f40d809e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60302/0 (socket says 172.21.15.62:60302) 2024-04-19T01:42:25.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40d7fff640 1 -- 172.21.15.62:0/2895629636 learned_addr learned my addr 172.21.15.62:0/2895629636 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:25.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40d7fff640 10 monclient(hunting): get_auth_request con 0x7f40d813cc60 auth_method 0 2024-04-19T01:42:25.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40d7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40cffff640 1 --2- 172.21.15.62:0/2895629636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40d8137780 0x7f40d81cfef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40d77fe640 1 --2- 172.21.15.62:0/2895629636 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f40d80a60b0 0x7f40d81cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40cffff640 10 monclient(hunting): get_auth_request con 0x7f40d8137780 auth_method 0 2024-04-19T01:42:25.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40cffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40cffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40d77fe640 10 monclient(hunting): get_auth_request con 0x7f40d80a60b0 auth_method 0 2024-04-19T01:42:25.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40d77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40d7fff640 10 monclient(hunting): handle_auth_done global_id 4235 payload 824 2024-04-19T01:42:25.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40d7fff640 1 -- 172.21.15.62:0/2895629636 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f40d80a60b0 msgr2=0x7f40d81cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40d7fff640 1 --2- 172.21.15.62:0/2895629636 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f40d80a60b0 0x7f40d81cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40d7fff640 1 -- 172.21.15.62:0/2895629636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40d8137780 msgr2=0x7f40d81cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40d7fff640 1 --2- 172.21.15.62:0/2895629636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40d8137780 0x7f40d81cfef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40d7fff640 10 monclient: _finish_hunting 0 2024-04-19T01:42:25.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40d7fff640 1 monclient: found mon.noname-c 2024-04-19T01:42:25.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:25.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40d7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f40d7fff640 1 -- 172.21.15.62:0/2895629636 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40d80a0500 con 0x7f40d813cc60 2024-04-19T01:42:25.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c16ffd640 4 set_mon_vals no callback set 2024-04-19T01:42:25.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c1f191640 10 monclient: _renew_subs 2024-04-19T01:42:25.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c1f191640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f3c1f191640 1 -- 172.21.15.62:0/2463928378 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c18141480 con 0x7f3c18142200 2024-04-19T01:42:25.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f84179c8640 1 -- 172.21.15.62:0/2332247771 wait complete. 2024-04-19T01:42:25.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f84179c8640 10 monclient: build_initial_monmap 2024-04-19T01:42:25.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.406+0000 7f84179c8640 10 monclient: monmap: 2024-04-19T01:42:25.476 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:25.476 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:25.476 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:25.476 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:25.476 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:25.476 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.476 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:25.476 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:25.476 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:25.477 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f84179c8640 1 Processor -- start 2024-04-19T01:42:25.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f3c1f191640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f3c1f191640 1 -- 172.21.15.62:0/2463928378 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3c18068ab0 con 0x7f3c18142200 2024-04-19T01:42:25.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f84179c8640 1 -- start start 2024-04-19T01:42:25.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f84179c8640 10 monclient: init 2024-04-19T01:42:25.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f38067fc640 1 --2- 172.21.15.62:0/1898297249 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f37f4036420 0x7f37f40388e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f380809bc20 tx=0x7f37fc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f3c157fa640 1 -- 172.21.15.62:0/2463928378 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c100b2040 con 0x7f3c18142200 2024-04-19T01:42:25.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f3c157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f3c157fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:25.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f3c157fa640 10 monclient: dump: 2024-04-19T01:42:25.478 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.478 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.478 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.478 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.478 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.478 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.479 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.479 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.479 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.479 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f3c157fa640 1 -- 172.21.15.62:0/2463928378 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f3c100c0020 con 0x7f3c18142200 2024-04-19T01:42:25.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f84179c8640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:25.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f84179c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-19T01:42:25.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f84179c8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84100a2360 0x7f84101dc740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f84179c8640 10 monclient(hunting): picked mon.noname-b con 0x7f84100a2360 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:25.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f84179c8640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f84100a2740 0x7f8410134340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f84179c8640 10 monclient(hunting): picked mon.noname-c con 0x7f84100a2740 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:25.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f84179c8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84101351b0 0x7f8410134880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f84179c8640 10 monclient(hunting): picked mon.noname-a con 0x7f84101351b0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:25.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f84179c8640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f84179c8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8410140080 con 0x7f84101351b0 2024-04-19T01:42:25.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f84179c8640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f84179c8640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f841013ff00 con 0x7f84100a2360 2024-04-19T01:42:25.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f84179c8640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f84179c8640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f8410140200 con 0x7f84100a2740 2024-04-19T01:42:25.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f84179c8640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:25.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f84179c8640 10 monclient(hunting): authenticate will time out at 931.232587s 2024-04-19T01:42:25.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f3c157fa640 1 --2- 172.21.15.62:0/2463928378 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f3c040363d0 0x7f3c04038890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f3c157fa640 1 -- 172.21.15.62:0/2463928378 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f3c100e43e0 con 0x7f3c18142200 2024-04-19T01:42:25.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8415f65640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84101351b0 0x7f8410134880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8415f65640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84101351b0 0x7f8410134880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36196/0 (socket says 172.21.15.62:36196) 2024-04-19T01:42:25.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8415f65640 1 -- 172.21.15.62:0/401555416 learned_addr learned my addr 172.21.15.62:0/401555416 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:25.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8415f65640 10 monclient(hunting): get_auth_request con 0x7f84101351b0 auth_method 0 2024-04-19T01:42:25.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8415f65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8415f65640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8415f65640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8415764640 1 --2- 172.21.15.62:0/401555416 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84100a2360 0x7f84101dc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8415764640 10 monclient(hunting): get_auth_request con 0x7f84100a2360 auth_method 0 2024-04-19T01:42:25.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8415764640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8415764640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8415764640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f3c157fa640 1 --2- 172.21.15.62:0/2463928378 >> [v2:172.21.15.62:6801/1128256055,v1:172.21.15.62:6803/1128256055] conn(0x7f3c0403c1d0 0x7f3c0403e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f3c157fa640 1 -- 172.21.15.62:0/2463928378 --> [v2:172.21.15.62:6801/1128256055,v1:172.21.15.62:6803/1128256055] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3c100aee50 con 0x7f3c0403c1d0 2024-04-19T01:42:25.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8415f65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8415f65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8414f63640 1 --2- 172.21.15.62:0/401555416 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f84100a2740 0x7f8410134340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8415f65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8414f63640 10 monclient(hunting): get_auth_request con 0x7f84100a2740 auth_method 0 2024-04-19T01:42:25.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8414f63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8414f63640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8414f63640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8414f63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8414f63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8414f63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f3c1cf2d640 1 --2- 172.21.15.62:0/2463928378 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f3c040363d0 0x7f3c04038890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8415764640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8415764640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8415764640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f3c1cf2d640 10 monclient: get_auth_request con 0x7f3c040363d0 auth_method 0 2024-04-19T01:42:25.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8415f65640 10 monclient(hunting): handle_auth_done global_id 4233 payload 824 2024-04-19T01:42:25.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8415f65640 1 -- 172.21.15.62:0/401555416 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f84100a2740 msgr2=0x7f8410134340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8415f65640 1 --2- 172.21.15.62:0/401555416 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f84100a2740 0x7f8410134340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8415f65640 1 -- 172.21.15.62:0/401555416 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84100a2360 msgr2=0x7f84101dc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8415f65640 1 --2- 172.21.15.62:0/401555416 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84100a2360 0x7f84101dc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8415f65640 10 monclient: _finish_hunting 0 2024-04-19T01:42:25.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8415f65640 1 monclient: found mon.noname-a 2024-04-19T01:42:25.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8415f65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:25.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8415f65640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:25.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8415f65640 1 -- 172.21.15.62:0/401555416 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f841013aac0 con 0x7f84101351b0 2024-04-19T01:42:25.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f3c17fff640 1 --2- 172.21.15.62:0/2463928378 >> [v2:172.21.15.62:6801/1128256055,v1:172.21.15.62:6803/1128256055] conn(0x7f3c0403c1d0 0x7f3c0403e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f3c17fff640 10 monclient: get_auth_request con 0x7f3c0403c1d0 auth_method 0 2024-04-19T01:42:25.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40d7fff640 1 -- 172.21.15.62:0/2895629636 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40d813cc60 msgr2=0x7f40d809e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:25.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40d7fff640 1 --2- 172.21.15.62:0/2895629636 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40d813cc60 0x7f40d809e860 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f40d000c780 tx=0x7f40d000cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40d4ff9640 1 -- 172.21.15.62:0/2895629636 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d0016020 con 0x7f40d813cc60 2024-04-19T01:42:25.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40d4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40d4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:25.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40d4ff9640 10 monclient: dump: 2024-04-19T01:42:25.488 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.488 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.488 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.488 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.488 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.488 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.488 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.488 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.489 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.489 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40d4ff9640 10 monclient: _finish_auth 0 2024-04-19T01:42:25.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40d4ff9640 10 monclient: _check_auth_tickets 2024-04-19T01:42:25.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40d4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:25.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40d4ff9640 1 -- 172.21.15.62:0/2895629636 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40d000ab80 con 0x7f40d813cc60 2024-04-19T01:42:25.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40d4ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:25.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40d4ff9640 1 -- 172.21.15.62:0/2895629636 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d0012420 con 0x7f40d813cc60 2024-04-19T01:42:25.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40ddc26640 5 monclient: authenticate success, global_id 4235 2024-04-19T01:42:25.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40ddc26640 10 monclient: _renew_subs 2024-04-19T01:42:25.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40ddc26640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40ddc26640 1 -- 172.21.15.62:0/2895629636 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40d80a1f50 con 0x7f40d813cc60 2024-04-19T01:42:25.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40d67fc640 4 set_mon_vals no callback set 2024-04-19T01:42:25.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40ddc26640 10 monclient: _renew_subs 2024-04-19T01:42:25.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40ddc26640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40ddc26640 1 -- 172.21.15.62:0/2895629636 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f40d81d0430 con 0x7f40d813cc60 2024-04-19T01:42:25.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f38077fe640 1 --2- 172.21.15.62:0/1898297249 >> [v2:172.21.15.71:6800/196330660,v1:172.21.15.71:6801/196330660] conn(0x7f37f403c290 0x7f37f403e6b0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40d4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40d4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:25.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40d4ff9640 10 monclient: dump: 2024-04-19T01:42:25.491 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.491 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.491 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.491 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.492 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.492 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.492 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.492 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.492 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.492 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40d4ff9640 1 -- 172.21.15.62:0/2895629636 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f40d0019020 con 0x7f40d813cc60 2024-04-19T01:42:25.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40d4ff9640 1 --2- 172.21.15.62:0/2895629636 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f40b8036470 0x7f40b8038930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40d4ff9640 1 -- 172.21.15.62:0/2895629636 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f40d0048ec0 con 0x7f40d813cc60 2024-04-19T01:42:25.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40ddc26640 1 --2- 172.21.15.62:0/2895629636 >> [v2:172.21.15.62:6816/199109455,v1:172.21.15.62:6817/199109455] conn(0x7f40d809eda0 0x7f40d809f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40ddc26640 1 -- 172.21.15.62:0/2895629636 --> [v2:172.21.15.62:6816/199109455,v1:172.21.15.62:6817/199109455] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f40d8068ab0 con 0x7f40d809eda0 2024-04-19T01:42:25.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40cffff640 1 --2- 172.21.15.62:0/2895629636 >> [v2:172.21.15.62:6816/199109455,v1:172.21.15.62:6817/199109455] conn(0x7f40d809eda0 0x7f40d809f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40cffff640 10 monclient: get_auth_request con 0x7f40d809eda0 auth_method 0 2024-04-19T01:42:25.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40cffff640 1 -- 172.21.15.62:0/2895629636 >> [v2:172.21.15.62:6816/199109455,v1:172.21.15.62:6817/199109455] conn(0x7f40d809eda0 msgr2=0x7f40d809f180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-19T01:42:25.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40cffff640 1 -- 172.21.15.62:0/2895629636 >> [v2:172.21.15.62:6816/199109455,v1:172.21.15.62:6817/199109455] conn(0x7f40d809eda0 msgr2=0x7f40d809f180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:42:25.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40cffff640 1 --2- 172.21.15.62:0/2895629636 >> [v2:172.21.15.62:6816/199109455,v1:172.21.15.62:6817/199109455] conn(0x7f40d809eda0 0x7f40d809f180 crc :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:42:25.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f40cffff640 1 --2- 172.21.15.62:0/2895629636 >> [v2:172.21.15.62:6816/199109455,v1:172.21.15.62:6817/199109455] conn(0x7f40d809eda0 0x7f40d809f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:42:25.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8415f65640 1 -- 172.21.15.62:0/401555416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84101351b0 msgr2=0x7f8410134880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:25.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8415f65640 1 --2- 172.21.15.62:0/401555416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84101351b0 0x7f8410134880 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f840800c780 tx=0x7f840800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8405ffb640 1 -- 172.21.15.62:0/401555416 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8408017070 con 0x7f84101351b0 2024-04-19T01:42:25.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8405ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8405ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:25.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8405ffb640 10 monclient: dump: 2024-04-19T01:42:25.494 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.494 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.494 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.494 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.494 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.494 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.494 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.494 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.494 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.494 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8405ffb640 10 monclient: _finish_auth 0 2024-04-19T01:42:25.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8405ffb640 10 monclient: _check_auth_tickets 2024-04-19T01:42:25.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8405ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:25.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8405ffb640 1 -- 172.21.15.62:0/401555416 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8408005070 con 0x7f84101351b0 2024-04-19T01:42:25.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8405ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:25.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8405ffb640 1 -- 172.21.15.62:0/401555416 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8408005460 con 0x7f84101351b0 2024-04-19T01:42:25.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8405ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8405ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:25.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f8405ffb640 10 monclient: dump: 2024-04-19T01:42:25.495 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.495 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.495 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.495 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.495 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.495 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.495 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.495 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.495 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.496 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f84077fe640 4 set_mon_vals no callback set 2024-04-19T01:42:25.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f3c1cf2d640 1 --2- 172.21.15.62:0/2463928378 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f3c040363d0 0x7f3c04038890 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c004fa0 tx=0x7f3c0c007ab0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f84179c8640 5 monclient: authenticate success, global_id 4233 2024-04-19T01:42:25.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f84179c8640 10 monclient: _renew_subs 2024-04-19T01:42:25.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f84179c8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:25.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f84179c8640 1 -- 172.21.15.62:0/401555416 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84100704d0 con 0x7f84101351b0 2024-04-19T01:42:25.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f84179c8640 10 monclient: _renew_subs 2024-04-19T01:42:25.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f84179c8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:25.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.410+0000 7f84179c8640 1 -- 172.21.15.62:0/401555416 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f841013cad0 con 0x7f84101351b0 2024-04-19T01:42:25.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.414+0000 7f37ef7fe640 1 -- 172.21.15.62:0/1898297249 <== osd.5 v2:172.21.15.71:6800/196330660 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f37f403ed60 con 0x7f37f403c290 2024-04-19T01:42:25.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.414+0000 7f40d77fe640 1 --2- 172.21.15.62:0/2895629636 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f40b8036470 0x7f40b8038930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.414+0000 7f84179c8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:25.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.414+0000 7f84179c8640 1 -- 172.21.15.62:0/401555416 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8410068ab0 con 0x7f84101351b0 2024-04-19T01:42:25.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.414+0000 7f40d77fe640 10 monclient: get_auth_request con 0x7f40b8036470 auth_method 0 2024-04-19T01:42:25.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.414+0000 7f8405ffb640 1 -- 172.21.15.62:0/401555416 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f840801a070 con 0x7f84101351b0 2024-04-19T01:42:25.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.414+0000 7f8405ffb640 1 --2- 172.21.15.62:0/401555416 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f83f4036470 0x7f83f4038930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.414+0000 7f8405ffb640 1 -- 172.21.15.62:0/401555416 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f8408048a20 con 0x7f84101351b0 2024-04-19T01:42:25.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.414+0000 7f8405ffb640 1 --2- 172.21.15.62:0/401555416 >> [v2:172.21.15.71:6816/2632024688,v1:172.21.15.71:6817/2632024688] conn(0x7f83f403c2f0 0x7f83f403e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.414+0000 7f8405ffb640 1 -- 172.21.15.62:0/401555416 --> [v2:172.21.15.71:6816/2632024688,v1:172.21.15.71:6817/2632024688] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f83f403edc0 con 0x7f83f403c2f0 2024-04-19T01:42:25.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.414+0000 7f8405ffb640 1 -- 172.21.15.62:0/401555416 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7f840804d2a0 con 0x7f84101351b0 2024-04-19T01:42:25.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.414+0000 7f8405ffb640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-04-19T01:42:25.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.414+0000 7f8414f63640 1 --2- 172.21.15.62:0/401555416 >> [v2:172.21.15.71:6816/2632024688,v1:172.21.15.71:6817/2632024688] conn(0x7f83f403c2f0 0x7f83f403e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.414+0000 7f8414f63640 10 monclient: get_auth_request con 0x7f83f403c2f0 auth_method 0 2024-04-19T01:42:25.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.414+0000 7f3c17fff640 1 --2- 172.21.15.62:0/2463928378 >> [v2:172.21.15.62:6801/1128256055,v1:172.21.15.62:6803/1128256055] conn(0x7f3c0403c1d0 0x7f3c0403e5f0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.414+0000 7f3c157fa640 1 -- 172.21.15.62:0/2463928378 <== osd.2 v2:172.21.15.62:6801/1128256055 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3c100aee50 con 0x7f3c0403c1d0 2024-04-19T01:42:25.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.414+0000 7f40d77fe640 1 --2- 172.21.15.62:0/2895629636 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f40b8036470 0x7f40b8038930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f40c80016c0 tx=0x7f40c8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.418+0000 7f8414f63640 1 --2- 172.21.15.62:0/401555416 >> [v2:172.21.15.71:6816/2632024688,v1:172.21.15.71:6817/2632024688] conn(0x7f83f403c2f0 0x7f83f403e710 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.422+0000 7f8405ffb640 1 -- 172.21.15.62:0/401555416 <== osd.3 v2:172.21.15.71:6816/2632024688 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f83f403edc0 con 0x7f83f403c2f0 2024-04-19T01:42:25.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.426+0000 7f380d6c8640 1 -- 172.21.15.62:0/1898297249 --> [v2:172.21.15.71:6800/196330660,v1:172.21.15.71:6801/196330660] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f37e8002c80 con 0x7f37f403c290 2024-04-19T01:42:25.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.426+0000 7f37ef7fe640 1 -- 172.21.15.62:0/1898297249 <== osd.5 v2:172.21.15.71:6800/196330660 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f37e8002c80 con 0x7f37f403c290 2024-04-19T01:42:25.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.426+0000 7f380d6c8640 1 -- 172.21.15.62:0/1898297249 >> [v2:172.21.15.71:6800/196330660,v1:172.21.15.71:6801/196330660] conn(0x7f37f403c290 msgr2=0x7f37f403e6b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.426+0000 7f380d6c8640 1 --2- 172.21.15.62:0/1898297249 >> [v2:172.21.15.71:6800/196330660,v1:172.21.15.71:6801/196330660] conn(0x7f37f403c290 0x7f37f403e6b0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.426+0000 7f380d6c8640 1 -- 172.21.15.62:0/1898297249 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f37f4036420 msgr2=0x7f37f40388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.426+0000 7f380d6c8640 1 --2- 172.21.15.62:0/1898297249 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f37f4036420 0x7f37f40388e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f380809bc20 tx=0x7f37fc008040 comp rx=0 tx=0).stop 2024-04-19T01:42:25.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.426+0000 7f380d6c8640 10 monclient: shutdown 2024-04-19T01:42:25.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.426+0000 7f380d6c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:25.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.426+0000 7f380d6c8640 1 -- 172.21.15.62:0/1898297249 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3808137780 msgr2=0x7f38081cf610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.426+0000 7f380d6c8640 1 --2- 172.21.15.62:0/1898297249 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3808137780 0x7f38081cf610 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f37f800c480 tx=0x7f37f800c940 comp rx=0 tx=0).stop 2024-04-19T01:42:25.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.426+0000 7f38077fe640 1 -- 172.21.15.62:0/1898297249 reap_dead start 2024-04-19T01:42:25.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.426+0000 7f380d6c8640 1 -- 172.21.15.62:0/1898297249 shutdown_connections 2024-04-19T01:42:25.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.426+0000 7f380d6c8640 1 -- 172.21.15.62:0/1898297249 >> 172.21.15.62:0/1898297249 conn(0x7f38080aa1d0 msgr2=0x7f38080a79d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:25.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.426+0000 7f380d6c8640 1 -- 172.21.15.62:0/1898297249 shutdown_connections 2024-04-19T01:42:25.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.426+0000 7f380d6c8640 1 -- 172.21.15.62:0/1898297249 wait complete. 2024-04-19T01:42:25.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.430+0000 7f3c1f191640 1 -- 172.21.15.62:0/2463928378 --> [v2:172.21.15.62:6801/1128256055,v1:172.21.15.62:6803/1128256055] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3c181d0330 con 0x7f3c0403c1d0 2024-04-19T01:42:25.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.430+0000 7f3c157fa640 1 -- 172.21.15.62:0/2463928378 <== osd.2 v2:172.21.15.62:6801/1128256055 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3c181d0330 con 0x7f3c0403c1d0 2024-04-19T01:42:25.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.430+0000 7f3c1f191640 1 -- 172.21.15.62:0/2463928378 >> [v2:172.21.15.62:6801/1128256055,v1:172.21.15.62:6803/1128256055] conn(0x7f3c0403c1d0 msgr2=0x7f3c0403e5f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.430+0000 7f3c1f191640 1 --2- 172.21.15.62:0/2463928378 >> [v2:172.21.15.62:6801/1128256055,v1:172.21.15.62:6803/1128256055] conn(0x7f3c0403c1d0 0x7f3c0403e5f0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.430+0000 7f3c1f191640 1 -- 172.21.15.62:0/2463928378 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f3c040363d0 msgr2=0x7f3c04038890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.430+0000 7f3c1f191640 1 --2- 172.21.15.62:0/2463928378 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f3c040363d0 0x7f3c04038890 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c004fa0 tx=0x7f3c0c007ab0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.430+0000 7f3c1f191640 10 monclient: shutdown 2024-04-19T01:42:25.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.430+0000 7f3c1f191640 20 monclient: shutdown canceling and discarding version request 1 2024-04-19T01:42:25.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.430+0000 7f3c1f191640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:25.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.430+0000 7f3c1f191640 1 -- 172.21.15.62:0/2463928378 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3c18142200 msgr2=0x7f3c18140d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.430+0000 7f3c1f191640 1 --2- 172.21.15.62:0/2463928378 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3c18142200 0x7f3c18140d70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3c100b0040 tx=0x7f3c100195d0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.430+0000 7f3c1d72e640 1 -- 172.21.15.62:0/2463928378 reap_dead start 2024-04-19T01:42:25.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.430+0000 7f3c157fa640 1 -- 172.21.15.62:0/2463928378 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (unknown 0 0 0) 0x7f3c100eb050 con 0x7f3c18142200 2024-04-19T01:42:25.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.430+0000 7f3c157fa640 10 monclient: discarding stray monitor message mon_get_version_reply(handle=1 version=15) 2024-04-19T01:42:25.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.430+0000 7f3c1f191640 1 -- 172.21.15.62:0/2463928378 shutdown_connections 2024-04-19T01:42:25.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.430+0000 7f3c1f191640 1 -- 172.21.15.62:0/2463928378 >> 172.21.15.62:0/2463928378 conn(0x7f3c180aa1d0 msgr2=0x7f3c181cfc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:25.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.430+0000 7f8415764640 1 --2- 172.21.15.62:0/401555416 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f83f4036470 0x7f83f4038930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.430+0000 7f8415764640 10 monclient: get_auth_request con 0x7f83f4036470 auth_method 0 2024-04-19T01:42:25.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.430+0000 7f3c1f191640 1 -- 172.21.15.62:0/2463928378 shutdown_connections 2024-04-19T01:42:25.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.430+0000 7f3c1f191640 1 -- 172.21.15.62:0/2463928378 wait complete. 2024-04-19T01:42:25.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.434+0000 7f8415764640 1 --2- 172.21.15.62:0/401555416 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f83f4036470 0x7f83f4038930 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f840c0098f0 tx=0x7f840c006f90 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.434+0000 7f84179c8640 1 -- 172.21.15.62:0/401555416 --> [v2:172.21.15.71:6816/2632024688,v1:172.21.15.71:6817/2632024688] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f841013b3a0 con 0x7f83f403c2f0 2024-04-19T01:42:25.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.434+0000 7f8405ffb640 1 -- 172.21.15.62:0/401555416 <== osd.3 v2:172.21.15.71:6816/2632024688 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f841013b3a0 con 0x7f83f403c2f0 2024-04-19T01:42:25.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.438+0000 7f84179c8640 1 -- 172.21.15.62:0/401555416 >> [v2:172.21.15.71:6816/2632024688,v1:172.21.15.71:6817/2632024688] conn(0x7f83f403c2f0 msgr2=0x7f83f403e710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.438+0000 7f84179c8640 1 --2- 172.21.15.62:0/401555416 >> [v2:172.21.15.71:6816/2632024688,v1:172.21.15.71:6817/2632024688] conn(0x7f83f403c2f0 0x7f83f403e710 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.438+0000 7f84179c8640 1 -- 172.21.15.62:0/401555416 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f83f4036470 msgr2=0x7f83f4038930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.438+0000 7f84179c8640 1 --2- 172.21.15.62:0/401555416 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f83f4036470 0x7f83f4038930 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f840c0098f0 tx=0x7f840c006f90 comp rx=0 tx=0).stop 2024-04-19T01:42:25.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.438+0000 7f84179c8640 10 monclient: shutdown 2024-04-19T01:42:25.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.438+0000 7f84179c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:25.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.438+0000 7f84179c8640 1 -- 172.21.15.62:0/401555416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84101351b0 msgr2=0x7f8410134880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.438+0000 7f84179c8640 1 --2- 172.21.15.62:0/401555416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84101351b0 0x7f8410134880 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f840800c780 tx=0x7f840800cc40 comp rx=0 tx=0).stop 2024-04-19T01:42:25.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.438+0000 7f8415f65640 1 -- 172.21.15.62:0/401555416 reap_dead start 2024-04-19T01:42:25.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.438+0000 7f84179c8640 1 -- 172.21.15.62:0/401555416 shutdown_connections 2024-04-19T01:42:25.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.438+0000 7f84179c8640 1 -- 172.21.15.62:0/401555416 >> 172.21.15.62:0/401555416 conn(0x7f84100a7070 msgr2=0x7f841006eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:25.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.438+0000 7f84179c8640 1 -- 172.21.15.62:0/401555416 shutdown_connections 2024-04-19T01:42:25.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.438+0000 7f84179c8640 1 -- 172.21.15.62:0/401555416 wait complete. 2024-04-19T01:42:25.514 INFO:teuthology.orchestra.run.smithi062.stdout:47244640259 2024-04-19T01:42:25.514 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-19T01:42:25.516 INFO:teuthology.orchestra.run.smithi062.stdout:47244640260 2024-04-19T01:42:25.516 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-19T01:42:25.517 INFO:teuthology.orchestra.run.smithi062.stdout:47244640259 2024-04-19T01:42:25.518 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-19T01:42:25.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f973688a640 10 monclient: get_monmap_and_config 2024-04-19T01:42:25.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f973688a640 10 monclient: build_initial_monmap 2024-04-19T01:42:25.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f973688a640 10 monclient: monmap: 2024-04-19T01:42:25.590 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:25.590 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.590 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:25.594675+0000 2024-04-19T01:42:25.590 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:25.594675+0000 2024-04-19T01:42:25.590 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:25.590 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.590 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:25.591 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:25.591 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:25.591 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:25.591 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:25.591 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f973688a640 1 Processor -- start 2024-04-19T01:42:25.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f973688a640 1 -- start start 2024-04-19T01:42:25.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f973688a640 10 monclient: init 2024-04-19T01:42:25.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f973688a640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:25.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f973688a640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-04-19T01:42:25.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f973688a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f97301355e0 addr v1:172.21.15.71:6789/0 2024-04-19T01:42:25.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f973688a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f97300726e0 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:25.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f973688a640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9730072b00 0x7f973013dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f973688a640 10 monclient(hunting): picked mon.noname-a con 0x7f9730072b00 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:25.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f973688a640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97300716c0 con 0x7f97300726e0 2024-04-19T01:42:25.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f973688a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97300719a0 con 0x7f97301355e0 2024-04-19T01:42:25.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f973688a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f973688a640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f97301360d0 con 0x7f9730072b00 2024-04-19T01:42:25.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f973688a640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:25.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f973688a640 10 monclient(hunting): authenticate will time out at 931.413852s 2024-04-19T01:42:25.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f972f7fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f97300726e0 0x7f973013a480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:33514/0 (socket says 172.21.15.62:33514) 2024-04-19T01:42:25.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f972f7fe640 1 -- 172.21.15.62:0/3559460888 learned_addr learned my addr 172.21.15.62:0/3559460888 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:42:25.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f9734e27640 1 --2- 172.21.15.62:0/3559460888 >> v2:172.21.15.62:3300/0 conn(0x7f9730072b00 0x7f973013dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f9734e27640 10 monclient(hunting): get_auth_request con 0x7f9730072b00 auth_method 0 2024-04-19T01:42:25.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f9734e27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f9734e27640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f9734e27640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f9734e27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f9734e27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f9734e27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f9734e27640 10 monclient(hunting): handle_auth_done global_id 4236 payload 274 2024-04-19T01:42:25.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f9734e27640 1 -- 172.21.15.62:0/3559460888 >> v1:172.21.15.71:6789/0 conn(0x7f97301355e0 legacy=0x7f97301359c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:25.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f9734e27640 1 -- 172.21.15.62:0/3559460888 >> v1:172.21.15.62:6789/0 conn(0x7f97300726e0 legacy=0x7f973013a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:25.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f9734e27640 10 monclient: _finish_hunting 0 2024-04-19T01:42:25.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f9734e27640 1 monclient: found mon.noname-a 2024-04-19T01:42:25.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f9734e27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:25.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f9734e27640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:25.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.590+0000 7f9734e27640 1 -- 172.21.15.62:0/3559460888 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9730070440 con 0x7f9730072b00 2024-04-19T01:42:25.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.594+0000 7f9734e27640 1 -- 172.21.15.62:0/3559460888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9730072b00 msgr2=0x7f973013dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:25.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.594+0000 7f9734e27640 1 --2- 172.21.15.62:0/3559460888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9730072b00 0x7f973013dbb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9728009d90 tx=0x7f972801c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52d379e228813ce2 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.594+0000 7f972effd640 1 -- 172.21.15.62:0/3559460888 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97280ac070 con 0x7f9730072b00 2024-04-19T01:42:25.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.594+0000 7f972effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.594+0000 7f972effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:25.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.594+0000 7f972effd640 10 monclient: dump: 2024-04-19T01:42:25.604 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.604 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.604 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.605 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.605 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.605 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.605 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.605 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.605 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.605 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.594+0000 7f972effd640 10 monclient: _finish_auth 0 2024-04-19T01:42:25.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.594+0000 7f972effd640 10 monclient: _check_auth_tickets 2024-04-19T01:42:25.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.594+0000 7f972effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:25.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.594+0000 7f972effd640 1 -- 172.21.15.62:0/3559460888 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97280a6760 con 0x7f9730072b00 2024-04-19T01:42:25.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.594+0000 7f972effd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:25.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.594+0000 7f973688a640 5 monclient: authenticate success, global_id 4236 2024-04-19T01:42:25.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.594+0000 7f972effd640 1 -- 172.21.15.62:0/3559460888 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97280a6b30 con 0x7f9730072b00 2024-04-19T01:42:25.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.594+0000 7f972effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.594+0000 7f972effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:25.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.594+0000 7f972effd640 10 monclient: dump: 2024-04-19T01:42:25.606 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.606 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.606 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.606 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.606 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.606 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.606 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.606 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.606 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.606 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.594+0000 7f973688a640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:25.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.594+0000 7f973688a640 4 set_mon_vals no callback set 2024-04-19T01:42:25.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.594+0000 7f973688a640 10 monclient: shutdown 2024-04-19T01:42:25.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.594+0000 7f973688a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:25.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.594+0000 7f973688a640 1 -- 172.21.15.62:0/3559460888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9730072b00 msgr2=0x7f973013dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.594+0000 7f973688a640 1 --2- 172.21.15.62:0/3559460888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9730072b00 0x7f973013dbb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9728009d90 tx=0x7f972801c830 comp rx=0 tx=0).stop 2024-04-19T01:42:25.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.594+0000 7f973688a640 1 -- 172.21.15.62:0/3559460888 shutdown_connections 2024-04-19T01:42:25.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.594+0000 7f973688a640 1 --2- 172.21.15.62:0/3559460888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9730072b00 0x7f973013dbb0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.594+0000 7f973688a640 1 -- 172.21.15.62:0/3559460888 >> 172.21.15.62:0/3559460888 conn(0x7f97300a8030 msgr2=0x7f97300a8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:25.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.594+0000 7f973688a640 1 -- 172.21.15.62:0/3559460888 shutdown_connections 2024-04-19T01:42:25.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.594+0000 7f973688a640 1 -- 172.21.15.62:0/3559460888 wait complete. 2024-04-19T01:42:25.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f973688a640 10 monclient: build_initial_monmap 2024-04-19T01:42:25.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f973688a640 10 monclient: monmap: 2024-04-19T01:42:25.607 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:25.607 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:25.608 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:25.608 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:25.608 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:25.608 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.608 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:25.608 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:25.608 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:25.608 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f973688a640 1 Processor -- start 2024-04-19T01:42:25.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f973688a640 1 -- start start 2024-04-19T01:42:25.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f973688a640 10 monclient: init 2024-04-19T01:42:25.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f973688a640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:25.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f973688a640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-19T01:42:25.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f973688a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97300726e0 0x7f973011e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f973688a640 10 monclient(hunting): picked mon.noname-c con 0x7f97300726e0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:25.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f973688a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9730072b00 0x7f973013b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f973688a640 10 monclient(hunting): picked mon.noname-a con 0x7f9730072b00 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:25.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f973688a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97301355e0 0x7f97301341e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f972ffff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97300726e0 0x7f973011e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f973688a640 10 monclient(hunting): picked mon.noname-b con 0x7f97301355e0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:25.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f973688a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f973688a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9730140ca0 con 0x7f9730072b00 2024-04-19T01:42:25.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f973688a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f972f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9730072b00 0x7f973013b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f972f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9730072b00 0x7f973013b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36222/0 (socket says 172.21.15.62:36222) 2024-04-19T01:42:25.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f972ffff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97300726e0 0x7f973011e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60320/0 (socket says 172.21.15.62:60320) 2024-04-19T01:42:25.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f972f7fe640 1 -- 172.21.15.62:0/2304285399 learned_addr learned my addr 172.21.15.62:0/2304285399 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:25.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f973688a640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f9730140b20 con 0x7f97301355e0 2024-04-19T01:42:25.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f973688a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f973688a640 1 -- 172.21.15.62:0/2304285399 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f9730140e20 con 0x7f97300726e0 2024-04-19T01:42:25.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f973688a640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:25.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f973688a640 10 monclient(hunting): authenticate will time out at 931.421649s 2024-04-19T01:42:25.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f972f7fe640 10 monclient(hunting): get_auth_request con 0x7f9730072b00 auth_method 0 2024-04-19T01:42:25.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f972f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f972f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f972f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f972ffff640 10 monclient(hunting): get_auth_request con 0x7f97300726e0 auth_method 0 2024-04-19T01:42:25.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f972ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f972ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f972ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f972f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f972f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f972f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f972ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f972ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f972ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f972f7fe640 10 monclient(hunting): handle_auth_done global_id 4239 payload 824 2024-04-19T01:42:25.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f972f7fe640 1 -- 172.21.15.62:0/2304285399 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97300726e0 msgr2=0x7f973011e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f972f7fe640 1 --2- 172.21.15.62:0/2304285399 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97300726e0 0x7f973011e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f972f7fe640 1 -- 172.21.15.62:0/2304285399 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97301355e0 msgr2=0x7f97301341e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f972f7fe640 1 --2- 172.21.15.62:0/2304285399 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97301355e0 0x7f97301341e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f972f7fe640 10 monclient: _finish_hunting 0 2024-04-19T01:42:25.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f972f7fe640 1 monclient: found mon.noname-a 2024-04-19T01:42:25.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f972f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:25.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f972f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:25.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.598+0000 7f972f7fe640 1 -- 172.21.15.62:0/2304285399 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f973013d550 con 0x7f9730072b00 2024-04-19T01:42:25.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.602+0000 7f972f7fe640 1 -- 172.21.15.62:0/2304285399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9730072b00 msgr2=0x7f973013b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:25.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.602+0000 7f972f7fe640 1 --2- 172.21.15.62:0/2304285399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9730072b00 0x7f973013b430 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f972400c470 tx=0x7f972400c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.602+0000 7f972cff9640 1 -- 172.21.15.62:0/2304285399 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9724007070 con 0x7f9730072b00 2024-04-19T01:42:25.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.602+0000 7f972cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.602+0000 7f972cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:25.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.602+0000 7f972cff9640 10 monclient: dump: 2024-04-19T01:42:25.612 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.612 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.612 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.612 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.612 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.612 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.612 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.612 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.612 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.612 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.602+0000 7f972cff9640 10 monclient: _finish_auth 0 2024-04-19T01:42:25.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.602+0000 7f972cff9640 10 monclient: _check_auth_tickets 2024-04-19T01:42:25.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.602+0000 7f972cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:25.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.602+0000 7f972cff9640 1 -- 172.21.15.62:0/2304285399 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9724007210 con 0x7f9730072b00 2024-04-19T01:42:25.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.602+0000 7f973688a640 5 monclient: authenticate success, global_id 4239 2024-04-19T01:42:25.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.602+0000 7f973688a640 10 monclient: _renew_subs 2024-04-19T01:42:25.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.602+0000 7f973688a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:25.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.602+0000 7f973688a640 1 -- 172.21.15.62:0/2304285399 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f973013cc10 con 0x7f9730072b00 2024-04-19T01:42:25.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.602+0000 7f972cff9640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:25.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.602+0000 7f972cff9640 1 -- 172.21.15.62:0/2304285399 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9724007600 con 0x7f9730072b00 2024-04-19T01:42:25.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.602+0000 7f972cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.602+0000 7f972cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:25.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.602+0000 7f972cff9640 10 monclient: dump: 2024-04-19T01:42:25.613 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.613 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.613 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.613 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.613 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.614 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.614 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.614 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.614 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.614 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.602+0000 7f972e7fc640 4 set_mon_vals no callback set 2024-04-19T01:42:25.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.602+0000 7f973688a640 10 monclient: _renew_subs 2024-04-19T01:42:25.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.602+0000 7f973688a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:25.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.602+0000 7f973688a640 1 -- 172.21.15.62:0/2304285399 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9730134720 con 0x7f9730072b00 2024-04-19T01:42:25.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.602+0000 7f973688a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:25.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.602+0000 7f973688a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:25.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.602+0000 7f973688a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:25.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.602+0000 7f973688a640 1 -- 172.21.15.62:0/2304285399 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9730068ab0 con 0x7f9730072b00 2024-04-19T01:42:25.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.602+0000 7f972cff9640 1 -- 172.21.15.62:0/2304285399 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f9724025430 con 0x7f9730072b00 2024-04-19T01:42:25.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.602+0000 7f972cff9640 1 --2- 172.21.15.62:0/2304285399 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f971c036420 0x7f971c0388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.602+0000 7f972cff9640 1 -- 172.21.15.62:0/2304285399 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f9724049ac0 con 0x7f9730072b00 2024-04-19T01:42:25.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.602+0000 7f972ffff640 1 --2- 172.21.15.62:0/2304285399 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f971c036420 0x7f971c0388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.602+0000 7f972ffff640 10 monclient: get_auth_request con 0x7f971c036420 auth_method 0 2024-04-19T01:42:25.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.606+0000 7f972ffff640 1 --2- 172.21.15.62:0/2304285399 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f971c036420 0x7f971c0388e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9720007fd0 tx=0x7f9720007e40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.606+0000 7f972cff9640 1 -- 172.21.15.62:0/2304285399 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f97240247e0 con 0x7f9730072b00 2024-04-19T01:42:25.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.606+0000 7f972cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:25.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.606+0000 7f972cff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:25.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.614+0000 7f40cffff640 1 --2- 172.21.15.62:0/2895629636 >> [v2:172.21.15.62:6816/199109455,v1:172.21.15.62:6817/199109455] conn(0x7f40d809eda0 0x7f40d809f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.614+0000 7f40cffff640 10 monclient: get_auth_request con 0x7f40d809eda0 auth_method 0 2024-04-19T01:42:25.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.618+0000 7f40cffff640 1 --2- 172.21.15.62:0/2895629636 >> [v2:172.21.15.62:6816/199109455,v1:172.21.15.62:6817/199109455] conn(0x7f40d809eda0 0x7f40d809f180 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.618+0000 7f40d4ff9640 1 -- 172.21.15.62:0/2895629636 <== osd.1 v2:172.21.15.62:6816/199109455 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f40d8068ab0 con 0x7f40d809eda0 2024-04-19T01:42:25.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0bea69c640 10 monclient: get_monmap_and_config 2024-04-19T01:42:25.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0bea69c640 10 monclient: build_initial_monmap 2024-04-19T01:42:25.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0bea69c640 10 monclient: monmap: 2024-04-19T01:42:25.623 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:25.623 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.623 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:25.627281+0000 2024-04-19T01:42:25.623 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:25.627281+0000 2024-04-19T01:42:25.623 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:25.623 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.623 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:25.623 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:25.623 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:25.623 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:25.623 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:25.623 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0bea69c640 1 Processor -- start 2024-04-19T01:42:25.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0bea69c640 1 -- start start 2024-04-19T01:42:25.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0bea69c640 10 monclient: init 2024-04-19T01:42:25.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0bea69c640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:25.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0bea69c640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-04-19T01:42:25.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0bea69c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0be41355e0 0x7f0be41359c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0bea69c640 10 monclient(hunting): picked mon.noname-c con 0x7f0be41355e0 addr v2:172.21.15.71:3300/0 2024-04-19T01:42:25.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0bea69c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0be40726e0 0x7f0be4072b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0bea69c640 10 monclient(hunting): picked mon.noname-a con 0x7f0be40726e0 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:25.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0bea69c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0be40a2970 0x7f0be40a2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be37fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0be40726e0 0x7f0be4072b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be37fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0be40726e0 0x7f0be4072b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36232/0 (socket says 172.21.15.62:36232) 2024-04-19T01:42:25.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be8c39640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0be40a2970 0x7f0be40a2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be8c39640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0be40a2970 0x7f0be40a2dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60326/0 (socket says 172.21.15.62:60326) 2024-04-19T01:42:25.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be37fe640 1 -- 172.21.15.62:0/1583404489 learned_addr learned my addr 172.21.15.62:0/1583404489 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:25.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0bea69c640 10 monclient(hunting): picked mon.noname-b con 0x7f0be40a2970 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:25.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0bea69c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0bea69c640 1 -- 172.21.15.62:0/1583404489 --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f0be40700b0 con 0x7f0be40726e0 2024-04-19T01:42:25.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0bea69c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0bea69c640 1 -- 172.21.15.62:0/1583404489 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f0be40716c0 con 0x7f0be41355e0 2024-04-19T01:42:25.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0bea69c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0bea69c640 1 -- 172.21.15.62:0/1583404489 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f0be40719a0 con 0x7f0be40a2970 2024-04-19T01:42:25.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0bea69c640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:25.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0bea69c640 10 monclient(hunting): authenticate will time out at 931.447330s 2024-04-19T01:42:25.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be37fe640 10 monclient(hunting): get_auth_request con 0x7f0be40726e0 auth_method 0 2024-04-19T01:42:25.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be8c39640 10 monclient(hunting): get_auth_request con 0x7f0be40a2970 auth_method 0 2024-04-19T01:42:25.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be8c39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be8c39640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be8c39640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be3fff640 1 --2- 172.21.15.62:0/1583404489 >> v2:172.21.15.71:3300/0 conn(0x7f0be41355e0 0x7f0be41359c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be3fff640 10 monclient(hunting): get_auth_request con 0x7f0be41355e0 auth_method 0 2024-04-19T01:42:25.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be8c39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be8c39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be8c39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be37fe640 10 monclient(hunting): handle_auth_done global_id 4242 payload 274 2024-04-19T01:42:25.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be37fe640 1 -- 172.21.15.62:0/1583404489 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0be40a2970 msgr2=0x7f0be40a2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:25.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be37fe640 1 --2- 172.21.15.62:0/1583404489 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0be40a2970 0x7f0be40a2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be37fe640 1 -- 172.21.15.62:0/1583404489 >> v2:172.21.15.71:3300/0 conn(0x7f0be41355e0 msgr2=0x7f0be41359c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:25.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be37fe640 1 --2- 172.21.15.62:0/1583404489 >> v2:172.21.15.71:3300/0 conn(0x7f0be41355e0 0x7f0be41359c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be37fe640 10 monclient: _finish_hunting 0 2024-04-19T01:42:25.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be37fe640 1 monclient: found mon.noname-a 2024-04-19T01:42:25.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:25.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:25.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be37fe640 1 -- 172.21.15.62:0/1583404489 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0be4070440 con 0x7f0be40726e0 2024-04-19T01:42:25.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.622+0000 7f0be8c39640 1 --2- 172.21.15.62:0/1583404489 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0be40a2970 0x7f0be40a2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:42:25.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.630+0000 7f0be37fe640 1 -- 172.21.15.62:0/1583404489 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0be40726e0 msgr2=0x7f0be4072b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:25.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.630+0000 7f0be37fe640 1 --2- 172.21.15.62:0/1583404489 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0be40726e0 0x7f0be4072b00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0be40a4440 tx=0x7f0bd80207f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=809b4db3a5a99fd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.630+0000 7f0be2ffd640 1 -- 172.21.15.62:0/1583404489 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd80ad480 con 0x7f0be40726e0 2024-04-19T01:42:25.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.630+0000 7f0be2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.630+0000 7f0be2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:25.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.630+0000 7f0be2ffd640 10 monclient: dump: 2024-04-19T01:42:25.632 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.632 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.632 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.632 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.632 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.632 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.632 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.632 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.632 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.632 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.630+0000 7f0be2ffd640 10 monclient: _finish_auth 0 2024-04-19T01:42:25.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.630+0000 7f0be2ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:42:25.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.630+0000 7f0be2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:25.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.630+0000 7f0be2ffd640 1 -- 172.21.15.62:0/1583404489 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bd80ad620 con 0x7f0be40726e0 2024-04-19T01:42:25.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.630+0000 7f0be2ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:25.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.630+0000 7f0be2ffd640 1 -- 172.21.15.62:0/1583404489 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd80ad9f0 con 0x7f0be40726e0 2024-04-19T01:42:25.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.630+0000 7f0be2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.630+0000 7f0be2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:25.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.630+0000 7f0be2ffd640 10 monclient: dump: 2024-04-19T01:42:25.633 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.633 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.633 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.633 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.633 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.633 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.634 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.634 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.634 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.634 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.630+0000 7f0bea69c640 5 monclient: authenticate success, global_id 4242 2024-04-19T01:42:25.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.630+0000 7f0bea69c640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:25.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.630+0000 7f0bea69c640 4 set_mon_vals no callback set 2024-04-19T01:42:25.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.630+0000 7f0bea69c640 10 monclient: shutdown 2024-04-19T01:42:25.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.630+0000 7f0bea69c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:25.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.630+0000 7f0bea69c640 1 -- 172.21.15.62:0/1583404489 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0be40726e0 msgr2=0x7f0be4072b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.630+0000 7f0bea69c640 1 --2- 172.21.15.62:0/1583404489 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0be40726e0 0x7f0be4072b00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0be40a4440 tx=0x7f0bd80207f0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.630+0000 7f0bea69c640 1 -- 172.21.15.62:0/1583404489 shutdown_connections 2024-04-19T01:42:25.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.630+0000 7f0bea69c640 1 --2- 172.21.15.62:0/1583404489 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0be40a2970 0x7f0be40a2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.630+0000 7f0bea69c640 1 --2- 172.21.15.62:0/1583404489 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0be40726e0 0x7f0be4072b00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.630+0000 7f0bea69c640 1 --2- 172.21.15.62:0/1583404489 >> v2:172.21.15.71:3300/0 conn(0x7f0be41355e0 0x7f0be41359c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.630+0000 7f0bea69c640 1 -- 172.21.15.62:0/1583404489 >> 172.21.15.62:0/1583404489 conn(0x7f0be40a8030 msgr2=0x7f0be40a8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:25.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.630+0000 7f0bea69c640 1 -- 172.21.15.62:0/1583404489 shutdown_connections 2024-04-19T01:42:25.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.630+0000 7f0bea69c640 1 -- 172.21.15.62:0/1583404489 wait complete. 2024-04-19T01:42:25.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0bea69c640 10 monclient: build_initial_monmap 2024-04-19T01:42:25.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0bea69c640 10 monclient: monmap: 2024-04-19T01:42:25.639 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:25.639 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:25.639 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:25.639 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:25.639 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:25.640 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.640 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:25.640 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:25.640 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:25.640 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0bea69c640 1 Processor -- start 2024-04-19T01:42:25.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0bea69c640 1 -- start start 2024-04-19T01:42:25.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0bea69c640 10 monclient: init 2024-04-19T01:42:25.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0bea69c640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:25.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0bea69c640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-19T01:42:25.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0bea69c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0be40726e0 0x7f0be412e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0bea69c640 10 monclient(hunting): picked mon.noname-c con 0x7f0be40726e0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:25.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0bea69c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0be40a2970 0x7f0be412ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be3fff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0be40726e0 0x7f0be412e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be3fff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0be40726e0 0x7f0be412e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60330/0 (socket says 172.21.15.62:60330) 2024-04-19T01:42:25.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0bea69c640 10 monclient(hunting): picked mon.noname-a con 0x7f0be40a2970 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:25.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be37fe640 1 --2- 172.21.15.62:0/3788694605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0be40a2970 0x7f0be412ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be3fff640 1 -- 172.21.15.62:0/3788694605 learned_addr learned my addr 172.21.15.62:0/3788694605 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:25.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0bea69c640 1 --2- 172.21.15.62:0/3788694605 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0be41355e0 0x7f0be409ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0bea69c640 10 monclient(hunting): picked mon.noname-b con 0x7f0be41355e0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:25.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0bea69c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0bea69c640 1 -- 172.21.15.62:0/3788694605 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0be413d030 con 0x7f0be40a2970 2024-04-19T01:42:25.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0bea69c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0bea69c640 1 -- 172.21.15.62:0/3788694605 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f0be413ceb0 con 0x7f0be41355e0 2024-04-19T01:42:25.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0bea69c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0bea69c640 1 -- 172.21.15.62:0/3788694605 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f0be413d1b0 con 0x7f0be40726e0 2024-04-19T01:42:25.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0bea69c640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:25.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0bea69c640 10 monclient(hunting): authenticate will time out at 931.458904s 2024-04-19T01:42:25.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be37fe640 10 monclient(hunting): get_auth_request con 0x7f0be40a2970 auth_method 0 2024-04-19T01:42:25.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be3fff640 10 monclient(hunting): get_auth_request con 0x7f0be40726e0 auth_method 0 2024-04-19T01:42:25.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f40ddc26640 1 -- 172.21.15.62:0/2895629636 --> [v2:172.21.15.62:6816/199109455,v1:172.21.15.62:6817/199109455] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f40d813f500 con 0x7f40d809eda0 2024-04-19T01:42:25.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be8c39640 1 --2- 172.21.15.62:0/3788694605 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0be41355e0 0x7f0be409ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be8c39640 10 monclient(hunting): get_auth_request con 0x7f0be41355e0 auth_method 0 2024-04-19T01:42:25.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be8c39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be8c39640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be8c39640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be8c39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be8c39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be8c39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f40d4ff9640 1 -- 172.21.15.62:0/2895629636 <== osd.1 v2:172.21.15.62:6816/199109455 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f40d813f500 con 0x7f40d809eda0 2024-04-19T01:42:25.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be37fe640 10 monclient(hunting): handle_auth_done global_id 4245 payload 824 2024-04-19T01:42:25.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be37fe640 1 -- 172.21.15.62:0/3788694605 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0be40726e0 msgr2=0x7f0be412e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be37fe640 1 --2- 172.21.15.62:0/3788694605 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0be40726e0 0x7f0be412e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be37fe640 1 -- 172.21.15.62:0/3788694605 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0be41355e0 msgr2=0x7f0be409ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be37fe640 1 --2- 172.21.15.62:0/3788694605 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0be41355e0 0x7f0be409ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be37fe640 10 monclient: _finish_hunting 0 2024-04-19T01:42:25.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be37fe640 1 monclient: found mon.noname-a 2024-04-19T01:42:25.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:25.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:25.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be37fe640 1 -- 172.21.15.62:0/3788694605 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0be40731b0 con 0x7f0be40a2970 2024-04-19T01:42:25.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be8c39640 1 --2- 172.21.15.62:0/3788694605 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0be41355e0 0x7f0be409ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:42:25.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.634+0000 7f0be3fff640 1 --2- 172.21.15.62:0/3788694605 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0be40726e0 0x7f0be412e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:42:25.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.638+0000 7f40ddc26640 1 -- 172.21.15.62:0/2895629636 >> [v2:172.21.15.62:6816/199109455,v1:172.21.15.62:6817/199109455] conn(0x7f40d809eda0 msgr2=0x7f40d809f180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.638+0000 7f40ddc26640 1 --2- 172.21.15.62:0/2895629636 >> [v2:172.21.15.62:6816/199109455,v1:172.21.15.62:6817/199109455] conn(0x7f40d809eda0 0x7f40d809f180 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.638+0000 7f40ddc26640 1 -- 172.21.15.62:0/2895629636 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f40b8036470 msgr2=0x7f40b8038930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.638+0000 7f40ddc26640 1 --2- 172.21.15.62:0/2895629636 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f40b8036470 0x7f40b8038930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f40c80016c0 tx=0x7f40c8009040 comp rx=0 tx=0).stop 2024-04-19T01:42:25.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.638+0000 7f40ddc26640 10 monclient: shutdown 2024-04-19T01:42:25.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.638+0000 7f40ddc26640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:25.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.638+0000 7f40ddc26640 1 -- 172.21.15.62:0/2895629636 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40d813cc60 msgr2=0x7f40d809e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.638+0000 7f40ddc26640 1 --2- 172.21.15.62:0/2895629636 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f40d813cc60 0x7f40d809e860 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f40d000c780 tx=0x7f40d000cc40 comp rx=0 tx=0).stop 2024-04-19T01:42:25.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.638+0000 7f40d7fff640 1 -- 172.21.15.62:0/2895629636 reap_dead start 2024-04-19T01:42:25.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.638+0000 7f40ddc26640 1 -- 172.21.15.62:0/2895629636 shutdown_connections 2024-04-19T01:42:25.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.638+0000 7f40ddc26640 1 -- 172.21.15.62:0/2895629636 >> 172.21.15.62:0/2895629636 conn(0x7f40d80aa1d0 msgr2=0x7f40d80a79d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:25.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.638+0000 7f40ddc26640 1 -- 172.21.15.62:0/2895629636 shutdown_connections 2024-04-19T01:42:25.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.638+0000 7f40ddc26640 1 -- 172.21.15.62:0/2895629636 wait complete. 2024-04-19T01:42:25.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.638+0000 7f0be37fe640 1 -- 172.21.15.62:0/3788694605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0be40a2970 msgr2=0x7f0be412ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:25.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.638+0000 7f0be37fe640 1 --2- 172.21.15.62:0/3788694605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0be40a2970 0x7f0be412ea50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0bd80b2000 tx=0x7f0bd801f030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.642+0000 7f0be0ff9640 1 -- 172.21.15.62:0/3788694605 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd80b5a10 con 0x7f0be40a2970 2024-04-19T01:42:25.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.642+0000 7f0be0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.642+0000 7f0be0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:25.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.642+0000 7f0be0ff9640 10 monclient: dump: 2024-04-19T01:42:25.650 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.650 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.651 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.651 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.651 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.651 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.651 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.651 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.651 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.651 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.642+0000 7f0be0ff9640 10 monclient: _finish_auth 0 2024-04-19T01:42:25.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.642+0000 7f0be0ff9640 10 monclient: _check_auth_tickets 2024-04-19T01:42:25.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.642+0000 7f0be0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:25.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.642+0000 7f0be0ff9640 1 -- 172.21.15.62:0/3788694605 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bd80b5bb0 con 0x7f0be40a2970 2024-04-19T01:42:25.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.642+0000 7f0be0ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:25.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.642+0000 7f0be0ff9640 1 -- 172.21.15.62:0/3788694605 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd80c0450 con 0x7f0be40a2970 2024-04-19T01:42:25.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.642+0000 7f0be0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.642+0000 7f0be0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:25.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.642+0000 7f0be0ff9640 10 monclient: dump: 2024-04-19T01:42:25.652 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.652 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.652 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.652 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.652 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.652 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.653 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.653 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.653 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.653 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.642+0000 7f0bea69c640 5 monclient: authenticate success, global_id 4245 2024-04-19T01:42:25.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.642+0000 7f0bea69c640 10 monclient: _renew_subs 2024-04-19T01:42:25.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.642+0000 7f0bea69c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:25.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.642+0000 7f0be27fc640 4 set_mon_vals no callback set 2024-04-19T01:42:25.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.642+0000 7f0bea69c640 1 -- 172.21.15.62:0/3788694605 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0be4131570 con 0x7f0be40a2970 2024-04-19T01:42:25.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.642+0000 7f0bea69c640 10 monclient: _renew_subs 2024-04-19T01:42:25.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.642+0000 7f0bea69c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:25.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.642+0000 7f0bea69c640 1 -- 172.21.15.62:0/3788694605 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0be41365b0 con 0x7f0be40a2970 2024-04-19T01:42:25.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.642+0000 7f0bea69c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:25.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.642+0000 7f0bea69c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:25.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.642+0000 7f0bea69c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:25.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.642+0000 7f0bea69c640 1 -- 172.21.15.62:0/3788694605 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0be4068ab0 con 0x7f0be40a2970 2024-04-19T01:42:25.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.642+0000 7f0be0ff9640 1 -- 172.21.15.62:0/3788694605 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f0bd80ab070 con 0x7f0be40a2970 2024-04-19T01:42:25.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.642+0000 7f0be0ff9640 1 --2- 172.21.15.62:0/3788694605 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0bd0036470 0x7f0bd0038930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.642+0000 7f0be0ff9640 1 -- 172.21.15.62:0/3788694605 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f0bd80e3850 con 0x7f0be40a2970 2024-04-19T01:42:25.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.642+0000 7f0be3fff640 1 --2- 172.21.15.62:0/3788694605 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0bd0036470 0x7f0bd0038930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.642+0000 7f0be3fff640 10 monclient: get_auth_request con 0x7f0bd0036470 auth_method 0 2024-04-19T01:42:25.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.646+0000 7f0be3fff640 1 --2- 172.21.15.62:0/3788694605 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0bd0036470 0x7f0bd0038930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0bd4009c20 tx=0x7f0bd4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.654+0000 7f0be0ff9640 1 -- 172.21.15.62:0/3788694605 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0bd800e070 con 0x7f0be40a2970 2024-04-19T01:42:25.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.654+0000 7f0be0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:25.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.654+0000 7f0be0ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:25.657 INFO:teuthology.orchestra.run.smithi062.stdout:47244640260 2024-04-19T01:42:25.658 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-19T01:42:25.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b97f1640 10 monclient: get_monmap_and_config 2024-04-19T01:42:25.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b97f1640 10 monclient: build_initial_monmap 2024-04-19T01:42:25.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b97f1640 10 monclient: monmap: 2024-04-19T01:42:25.684 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:25.684 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.684 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:25.688332+0000 2024-04-19T01:42:25.684 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:25.688332+0000 2024-04-19T01:42:25.684 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:25.684 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.684 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:25.684 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:25.684 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:25.684 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:25.685 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:25.685 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b97f1640 1 Processor -- start 2024-04-19T01:42:25.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b97f1640 1 -- start start 2024-04-19T01:42:25.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b97f1640 10 monclient: init 2024-04-19T01:42:25.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b97f1640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:25.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b97f1640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-04-19T01:42:25.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b97f1640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe7b4137780 0x7fe7b4137b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b97f1640 10 monclient(hunting): picked mon.noname-a con 0x7fe7b4137780 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:25.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b97f1640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe7b41380a0 0x7fe7b40a45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b97f1640 10 monclient(hunting): picked mon.noname-c con 0x7fe7b41380a0 addr v2:172.21.15.71:3300/0 2024-04-19T01:42:25.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b97f1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7b40a60b0 0x7fe7b40a4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b97f1640 10 monclient(hunting): picked mon.noname-b con 0x7fe7b40a60b0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:25.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b97f1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b97f1640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fe7b409bd20 con 0x7fe7b4137780 2024-04-19T01:42:25.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b97f1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b97f1640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fe7b409c2e0 con 0x7fe7b41380a0 2024-04-19T01:42:25.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b97f1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b97f1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fe7b409c000 con 0x7fe7b40a60b0 2024-04-19T01:42:25.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b97f1640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:25.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b97f1640 10 monclient(hunting): authenticate will time out at 931.507579s 2024-04-19T01:42:25.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b2ffd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe7b4137780 0x7fe7b4137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b2ffd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe7b4137780 0x7fe7b4137b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36254/0 (socket says 172.21.15.62:36254) 2024-04-19T01:42:25.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b2ffd640 1 -- 172.21.15.62:0/401635700 learned_addr learned my addr 172.21.15.62:0/401635700 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:25.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b2ffd640 10 monclient(hunting): get_auth_request con 0x7fe7b4137780 auth_method 0 2024-04-19T01:42:25.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b37fe640 1 --2- 172.21.15.62:0/401635700 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7b40a60b0 0x7fe7b40a4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b37fe640 10 monclient(hunting): get_auth_request con 0x7fe7b40a60b0 auth_method 0 2024-04-19T01:42:25.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.682+0000 7fe7b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.686+0000 7fe7b27fc640 1 --2- 172.21.15.62:0/401635700 >> v2:172.21.15.71:3300/0 conn(0x7fe7b41380a0 0x7fe7b40a45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.686+0000 7fe7b27fc640 10 monclient(hunting): get_auth_request con 0x7fe7b41380a0 auth_method 0 2024-04-19T01:42:25.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.686+0000 7fe7b27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.686+0000 7fe7b27fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.686+0000 7fe7b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.686+0000 7fe7b37fe640 10 monclient(hunting): handle_auth_done global_id 4247 payload 274 2024-04-19T01:42:25.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.686+0000 7fe7b37fe640 1 -- 172.21.15.62:0/401635700 >> v2:172.21.15.71:3300/0 conn(0x7fe7b41380a0 msgr2=0x7fe7b40a45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:25.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.686+0000 7fe7b37fe640 1 --2- 172.21.15.62:0/401635700 >> v2:172.21.15.71:3300/0 conn(0x7fe7b41380a0 0x7fe7b40a45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.686+0000 7fe7b37fe640 1 -- 172.21.15.62:0/401635700 >> v2:172.21.15.62:3300/0 conn(0x7fe7b4137780 msgr2=0x7fe7b4137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:25.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.686+0000 7fe7b37fe640 1 --2- 172.21.15.62:0/401635700 >> v2:172.21.15.62:3300/0 conn(0x7fe7b4137780 0x7fe7b4137b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.686+0000 7fe7b37fe640 10 monclient: _finish_hunting 0 2024-04-19T01:42:25.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.686+0000 7fe7b37fe640 1 monclient: found mon.noname-b 2024-04-19T01:42:25.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.686+0000 7fe7b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:25.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.686+0000 7fe7b37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.686+0000 7fe7b37fe640 1 -- 172.21.15.62:0/401635700 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7b4073210 con 0x7fe7b40a60b0 2024-04-19T01:42:25.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.690+0000 7fe7b37fe640 1 -- 172.21.15.62:0/401635700 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7b40a60b0 msgr2=0x7fe7b40a4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:25.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.690+0000 7fe7b37fe640 1 --2- 172.21.15.62:0/401635700 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7b40a60b0 0x7fe7b40a4b10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe7b40a74f0 tx=0x7fe7ac01c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83299abfb4390bf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.690+0000 7fe7b1ffb640 1 -- 172.21.15.62:0/401635700 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7ac0b2070 con 0x7fe7b40a60b0 2024-04-19T01:42:25.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.690+0000 7fe7b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.690+0000 7fe7b1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:42:25.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.690+0000 7fe7b1ffb640 10 monclient: dump: 2024-04-19T01:42:25.693 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.693 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.693 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.693 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.693 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.693 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.694 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.694 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.694 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.694 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.690+0000 7fe7b1ffb640 10 monclient: _finish_auth 0 2024-04-19T01:42:25.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.690+0000 7fe7b1ffb640 10 monclient: _check_auth_tickets 2024-04-19T01:42:25.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.690+0000 7fe7b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:25.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.690+0000 7fe7b1ffb640 1 -- 172.21.15.62:0/401635700 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7ac0a5780 con 0x7fe7b40a60b0 2024-04-19T01:42:25.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.690+0000 7fe7b97f1640 5 monclient: authenticate success, global_id 4247 2024-04-19T01:42:25.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.690+0000 7fe7b97f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:42:25.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.690+0000 7fe7b1ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:25.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.690+0000 7fe7b97f1640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:25.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.690+0000 7fe7b1ffb640 1 -- 172.21.15.62:0/401635700 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7ac0a5b50 con 0x7fe7b40a60b0 2024-04-19T01:42:25.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.690+0000 7fe7b97f1640 4 set_mon_vals no callback set 2024-04-19T01:42:25.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.690+0000 7fe7b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.690+0000 7fe7b1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:25.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.690+0000 7fe7b1ffb640 10 monclient: dump: 2024-04-19T01:42:25.695 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.695 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.695 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.696 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.696 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.696 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.696 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.696 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.696 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.696 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.690+0000 7fe7b97f1640 10 monclient: shutdown 2024-04-19T01:42:25.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.690+0000 7fe7b97f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:25.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.690+0000 7fe7b97f1640 1 -- 172.21.15.62:0/401635700 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7b40a60b0 msgr2=0x7fe7b40a4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.690+0000 7fe7b97f1640 1 --2- 172.21.15.62:0/401635700 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7b40a60b0 0x7fe7b40a4b10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe7b40a74f0 tx=0x7fe7ac01c7f0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.690+0000 7fe7b97f1640 1 -- 172.21.15.62:0/401635700 shutdown_connections 2024-04-19T01:42:25.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.690+0000 7fe7b97f1640 1 --2- 172.21.15.62:0/401635700 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7b40a60b0 0x7fe7b40a4b10 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.690+0000 7fe7b97f1640 1 --2- 172.21.15.62:0/401635700 >> v2:172.21.15.71:3300/0 conn(0x7fe7b41380a0 0x7fe7b40a45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.690+0000 7fe7b97f1640 1 --2- 172.21.15.62:0/401635700 >> v2:172.21.15.62:3300/0 conn(0x7fe7b4137780 0x7fe7b4137b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.690+0000 7fe7b97f1640 1 -- 172.21.15.62:0/401635700 >> 172.21.15.62:0/401635700 conn(0x7fe7b40aa1d0 msgr2=0x7fe7b40a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:25.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.690+0000 7fe7b97f1640 1 -- 172.21.15.62:0/401635700 shutdown_connections 2024-04-19T01:42:25.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.690+0000 7fe7b97f1640 1 -- 172.21.15.62:0/401635700 wait complete. 2024-04-19T01:42:25.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.690+0000 7fe7b97f1640 10 monclient: build_initial_monmap 2024-04-19T01:42:25.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.690+0000 7fe7b97f1640 10 monclient: monmap: 2024-04-19T01:42:25.697 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:25.697 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:25.697 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:25.697 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:25.697 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:25.697 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.697 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:25.697 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:25.697 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:25.697 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b97f1640 1 Processor -- start 2024-04-19T01:42:25.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b97f1640 1 -- start start 2024-04-19T01:42:25.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b97f1640 10 monclient: init 2024-04-19T01:42:25.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b97f1640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:25.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b97f1640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-19T01:42:25.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b97f1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7b40a60b0 0x7fe7b41cf400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b97f1640 10 monclient(hunting): picked mon.noname-c con 0x7fe7b40a60b0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:25.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b97f1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe7b4137780 0x7fe7b4136450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b97f1640 10 monclient(hunting): picked mon.noname-b con 0x7fe7b4137780 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:25.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b97f1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7b41380a0 0x7fe7b40a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b97f1640 10 monclient(hunting): picked mon.noname-a con 0x7fe7b41380a0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:25.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b97f1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b97f1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe7b4141160 con 0x7fe7b41380a0 2024-04-19T01:42:25.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b97f1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b97f1640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe7b40a5460 con 0x7fe7b4137780 2024-04-19T01:42:25.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b97f1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b97f1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fe7b41412e0 con 0x7fe7b40a60b0 2024-04-19T01:42:25.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b97f1640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:25.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b97f1640 10 monclient(hunting): authenticate will time out at 931.516845s 2024-04-19T01:42:25.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b37fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7b41380a0 0x7fe7b40a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b37fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7b41380a0 0x7fe7b40a0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36256/0 (socket says 172.21.15.62:36256) 2024-04-19T01:42:25.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b37fe640 1 -- 172.21.15.62:0/2034685977 learned_addr learned my addr 172.21.15.62:0/2034685977 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:25.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b37fe640 10 monclient(hunting): get_auth_request con 0x7fe7b41380a0 auth_method 0 2024-04-19T01:42:25.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b2ffd640 1 --2- 172.21.15.62:0/2034685977 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7b40a60b0 0x7fe7b41cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b2ffd640 10 monclient(hunting): get_auth_request con 0x7fe7b40a60b0 auth_method 0 2024-04-19T01:42:25.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b27fc640 1 --2- 172.21.15.62:0/2034685977 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe7b4137780 0x7fe7b4136450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b27fc640 10 monclient(hunting): get_auth_request con 0x7fe7b4137780 auth_method 0 2024-04-19T01:42:25.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b27fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b37fe640 10 monclient(hunting): handle_auth_done global_id 4251 payload 824 2024-04-19T01:42:25.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b37fe640 1 -- 172.21.15.62:0/2034685977 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7b40a60b0 msgr2=0x7fe7b41cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b37fe640 1 --2- 172.21.15.62:0/2034685977 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7b40a60b0 0x7fe7b41cf400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b37fe640 1 -- 172.21.15.62:0/2034685977 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe7b4137780 msgr2=0x7fe7b4136450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b37fe640 1 --2- 172.21.15.62:0/2034685977 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe7b4137780 0x7fe7b4136450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b37fe640 10 monclient: _finish_hunting 0 2024-04-19T01:42:25.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b37fe640 1 monclient: found mon.noname-a 2024-04-19T01:42:25.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:25.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:25.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b37fe640 1 -- 172.21.15.62:0/2034685977 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7b413ddd0 con 0x7fe7b41380a0 2024-04-19T01:42:25.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b27fc640 1 --2- 172.21.15.62:0/2034685977 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe7b4137780 0x7fe7b4136450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T01:42:25.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.694+0000 7fe7b37fe640 1 -- 172.21.15.62:0/2034685977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7b41380a0 msgr2=0x7fe7b40a0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:25.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fe7b37fe640 1 --2- 172.21.15.62:0/2034685977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7b41380a0 0x7fe7b40a0850 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac01aea0 tx=0x7fe7ac019590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fe79b7fe640 1 -- 172.21.15.62:0/2034685977 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7ac0bd020 con 0x7fe7b41380a0 2024-04-19T01:42:25.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fe79b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fe79b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:25.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fe79b7fe640 10 monclient: dump: 2024-04-19T01:42:25.704 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.704 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.704 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.705 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.705 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.705 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.705 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.705 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.705 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.705 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fe79b7fe640 10 monclient: _finish_auth 0 2024-04-19T01:42:25.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fe79b7fe640 10 monclient: _check_auth_tickets 2024-04-19T01:42:25.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fe79b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:25.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fe79b7fe640 1 -- 172.21.15.62:0/2034685977 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7ac0ae990 con 0x7fe7b41380a0 2024-04-19T01:42:25.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fe79b7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:25.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fe79b7fe640 1 -- 172.21.15.62:0/2034685977 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7ac0b2040 con 0x7fe7b41380a0 2024-04-19T01:42:25.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fe79b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fe79b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:25.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fe79b7fe640 10 monclient: dump: 2024-04-19T01:42:25.706 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.706 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.706 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.706 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.706 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.706 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.706 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.706 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.706 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.706 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fe7b17fa640 4 set_mon_vals no callback set 2024-04-19T01:42:25.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fe7b97f1640 5 monclient: authenticate success, global_id 4251 2024-04-19T01:42:25.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fe7b97f1640 10 monclient: _renew_subs 2024-04-19T01:42:25.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fe7b97f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:25.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fe7b97f1640 1 -- 172.21.15.62:0/2034685977 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7b41d0280 con 0x7fe7b41380a0 2024-04-19T01:42:25.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fe7b97f1640 10 monclient: _renew_subs 2024-04-19T01:42:25.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fe7b97f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:25.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fe7b97f1640 1 -- 172.21.15.62:0/2034685977 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7b4121df0 con 0x7fe7b41380a0 2024-04-19T01:42:25.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fe7b97f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:25.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fe7b97f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:25.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fe7b97f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:25.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fe7b97f1640 1 -- 172.21.15.62:0/2034685977 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7b4068ab0 con 0x7fe7b41380a0 2024-04-19T01:42:25.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fe79b7fe640 1 -- 172.21.15.62:0/2034685977 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fe7ac0c0020 con 0x7fe7b41380a0 2024-04-19T01:42:25.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fe79b7fe640 1 --2- 172.21.15.62:0/2034685977 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fe7a0036470 0x7fe7a0038930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fe79b7fe640 1 -- 172.21.15.62:0/2034685977 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7fe7ac0e41e0 con 0x7fe7b41380a0 2024-04-19T01:42:25.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fe7b2ffd640 1 --2- 172.21.15.62:0/2034685977 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fe7a0036470 0x7fe7a0038930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fe7b2ffd640 10 monclient: get_auth_request con 0x7fe7a0036470 auth_method 0 2024-04-19T01:42:25.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fdebdeb1640 10 monclient: get_monmap_and_config 2024-04-19T01:42:25.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fdebdeb1640 10 monclient: build_initial_monmap 2024-04-19T01:42:25.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fdebdeb1640 10 monclient: monmap: 2024-04-19T01:42:25.710 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:25.710 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.710 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:25.705405+0000 2024-04-19T01:42:25.710 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:25.705405+0000 2024-04-19T01:42:25.710 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:25.710 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.710 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:25.710 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:25.710 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:25.711 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:25.711 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:25.711 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.698+0000 7fdebdeb1640 1 Processor -- start 2024-04-19T01:42:25.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdebdeb1640 1 -- start start 2024-04-19T01:42:25.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdebdeb1640 10 monclient: init 2024-04-19T01:42:25.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdebdeb1640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:25.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdebdeb1640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-04-19T01:42:25.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdebdeb1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdeb8137780 addr v1:172.21.15.71:6789/0 2024-04-19T01:42:25.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdebdeb1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdeb80a60b0 0x7fdeb813c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdebdeb1640 10 monclient(hunting): picked mon.noname-b con 0x7fdeb80a60b0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:25.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdebdeb1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdeb813cc60 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:25.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdeb6ffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdeb80a60b0 0x7fdeb813c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdeb7fff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fdeb813cc60 0x7fdeb813f050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:33522/0 (socket says 172.21.15.62:33522) 2024-04-19T01:42:25.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdeb6ffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdeb80a60b0 0x7fdeb813c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60350/0 (socket says 172.21.15.62:60350) 2024-04-19T01:42:25.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdeb7fff640 1 -- 172.21.15.62:0/3744597756 learned_addr learned my addr 172.21.15.62:0/3744597756 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:42:25.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdebdeb1640 1 -- 172.21.15.62:0/3744597756 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdeb809c2e0 con 0x7fdeb813cc60 2024-04-19T01:42:25.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdebdeb1640 1 -- 172.21.15.62:0/3744597756 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdeb809c000 con 0x7fdeb8137780 2024-04-19T01:42:25.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdebdeb1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdebdeb1640 1 -- 172.21.15.62:0/3744597756 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fdeb809bd20 con 0x7fdeb80a60b0 2024-04-19T01:42:25.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdebdeb1640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:25.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdebdeb1640 10 monclient(hunting): authenticate will time out at 931.525415s 2024-04-19T01:42:25.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdeb6ffd640 10 monclient(hunting): get_auth_request con 0x7fdeb80a60b0 auth_method 0 2024-04-19T01:42:25.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdeb6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdeb6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdeb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdeb6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdeb6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdeb6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdeb6ffd640 10 monclient(hunting): handle_auth_done global_id 4253 payload 274 2024-04-19T01:42:25.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdeb6ffd640 1 -- 172.21.15.62:0/3744597756 >> v1:172.21.15.71:6789/0 conn(0x7fdeb8137780 legacy=0x7fdeb8137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:25.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdeb6ffd640 1 -- 172.21.15.62:0/3744597756 >> v1:172.21.15.62:6789/0 conn(0x7fdeb813cc60 legacy=0x7fdeb813f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:25.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdeb6ffd640 10 monclient: _finish_hunting 0 2024-04-19T01:42:25.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdeb6ffd640 1 monclient: found mon.noname-b 2024-04-19T01:42:25.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdeb6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:25.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdeb6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fdeb6ffd640 1 -- 172.21.15.62:0/3744597756 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeb8138270 con 0x7fdeb80a60b0 2024-04-19T01:42:25.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.702+0000 7fe7b2ffd640 1 --2- 172.21.15.62:0/2034685977 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fe7a0036470 0x7fe7a0038930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4006cd0 tx=0x7fe7a4009240 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.706+0000 7fe79b7fe640 1 -- 172.21.15.62:0/2034685977 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe7ac0c7030 con 0x7fe7b41380a0 2024-04-19T01:42:25.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.706+0000 7fe79b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:25.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.706+0000 7fe79b7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:25.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.706+0000 7fdeb6ffd640 1 -- 172.21.15.62:0/3744597756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdeb80a60b0 msgr2=0x7fdeb813c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:25.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.706+0000 7fdeb6ffd640 1 --2- 172.21.15.62:0/3744597756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdeb80a60b0 0x7fdeb813c720 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fdeac009820 tx=0x7fdeac01c1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b62d73ad6b5800b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.706+0000 7fdeb67fc640 1 -- 172.21.15.62:0/3744597756 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeac0b0070 con 0x7fdeb80a60b0 2024-04-19T01:42:25.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.706+0000 7fdeb67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.706+0000 7fdeb67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:42:25.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.706+0000 7fdeb67fc640 10 monclient: dump: 2024-04-19T01:42:25.716 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.716 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.716 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.716 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.716 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.716 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.716 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.716 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.716 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.716 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.706+0000 7fdeb67fc640 10 monclient: _finish_auth 0 2024-04-19T01:42:25.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.706+0000 7fdeb67fc640 10 monclient: _check_auth_tickets 2024-04-19T01:42:25.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.706+0000 7fdeb67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:25.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.706+0000 7fdeb67fc640 1 -- 172.21.15.62:0/3744597756 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdeac0ab600 con 0x7fdeb80a60b0 2024-04-19T01:42:25.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.706+0000 7fdebdeb1640 5 monclient: authenticate success, global_id 4253 2024-04-19T01:42:25.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.706+0000 7fdebdeb1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:42:25.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.706+0000 7fdeb67fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:25.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.706+0000 7fdebdeb1640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:25.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.706+0000 7fdebdeb1640 4 set_mon_vals no callback set 2024-04-19T01:42:25.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.706+0000 7fdeb67fc640 1 -- 172.21.15.62:0/3744597756 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeac0ab9d0 con 0x7fdeb80a60b0 2024-04-19T01:42:25.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.706+0000 7fdeb67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.706+0000 7fdeb67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:25.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.706+0000 7fdeb67fc640 10 monclient: dump: 2024-04-19T01:42:25.717 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.717 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.717 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.717 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.717 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.718 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.718 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.718 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.718 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.718 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.706+0000 7fdebdeb1640 10 monclient: shutdown 2024-04-19T01:42:25.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.706+0000 7fdebdeb1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:25.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.706+0000 7fdebdeb1640 1 -- 172.21.15.62:0/3744597756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdeb80a60b0 msgr2=0x7fdeb813c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.706+0000 7fdebdeb1640 1 --2- 172.21.15.62:0/3744597756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdeb80a60b0 0x7fdeb813c720 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fdeac009820 tx=0x7fdeac01c1a0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.706+0000 7fdebdeb1640 1 -- 172.21.15.62:0/3744597756 shutdown_connections 2024-04-19T01:42:25.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.706+0000 7fdebdeb1640 1 --2- 172.21.15.62:0/3744597756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdeb80a60b0 0x7fdeb813c720 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.706+0000 7fdebdeb1640 1 -- 172.21.15.62:0/3744597756 >> 172.21.15.62:0/3744597756 conn(0x7fdeb80aa1d0 msgr2=0x7fdeb80a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:25.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdebdeb1640 1 -- 172.21.15.62:0/3744597756 shutdown_connections 2024-04-19T01:42:25.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdebdeb1640 1 -- 172.21.15.62:0/3744597756 wait complete. 2024-04-19T01:42:25.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d737d640 10 monclient: get_monmap_and_config 2024-04-19T01:42:25.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d737d640 10 monclient: build_initial_monmap 2024-04-19T01:42:25.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdebdeb1640 10 monclient: build_initial_monmap 2024-04-19T01:42:25.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdebdeb1640 10 monclient: monmap: 2024-04-19T01:42:25.719 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:25.719 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:25.720 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:25.720 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:25.720 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:25.720 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.720 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:25.720 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:25.720 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:25.720 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d737d640 10 monclient: monmap: 2024-04-19T01:42:25.720 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:25.720 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.720 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:25.714556+0000 2024-04-19T01:42:25.721 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:25.714556+0000 2024-04-19T01:42:25.721 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:25.721 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.721 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:25.721 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:25.721 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:25.721 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:25.721 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:25.721 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdebdeb1640 1 Processor -- start 2024-04-19T01:42:25.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdebdeb1640 1 -- start start 2024-04-19T01:42:25.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdebdeb1640 10 monclient: init 2024-04-19T01:42:25.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d737d640 1 Processor -- start 2024-04-19T01:42:25.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d737d640 1 -- start start 2024-04-19T01:42:25.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d737d640 10 monclient: init 2024-04-19T01:42:25.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdebdeb1640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:25.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdebdeb1640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-19T01:42:25.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdebdeb1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdeb80a60b0 0x7fdeb81cfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdebdeb1640 10 monclient(hunting): picked mon.noname-b con 0x7fdeb80a60b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:25.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdebdeb1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdeb8137780 0x7fdeb809e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdebdeb1640 10 monclient(hunting): picked mon.noname-c con 0x7fdeb8137780 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:25.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdebdeb1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdeb813cc60 0x7fdeb809eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdebdeb1640 10 monclient(hunting): picked mon.noname-a con 0x7fdeb813cc60 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:25.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdebdeb1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdebdeb1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb8143320 con 0x7fdeb813cc60 2024-04-19T01:42:25.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdebdeb1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdebdeb1640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb81431a0 con 0x7fdeb80a60b0 2024-04-19T01:42:25.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdebdeb1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdebdeb1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fdeb81434a0 con 0x7fdeb8137780 2024-04-19T01:42:25.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdebdeb1640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:25.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdebdeb1640 10 monclient(hunting): authenticate will time out at 931.533558s 2024-04-19T01:42:25.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdeb6ffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdeb8137780 0x7fdeb809e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdeb6ffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdeb8137780 0x7fdeb809e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60352/0 (socket says 172.21.15.62:60352) 2024-04-19T01:42:25.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdeb6ffd640 1 -- 172.21.15.62:0/4012220678 learned_addr learned my addr 172.21.15.62:0/4012220678 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:25.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdeb6ffd640 10 monclient(hunting): get_auth_request con 0x7fdeb8137780 auth_method 0 2024-04-19T01:42:25.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdeb6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdeb6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdeb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdeb7fff640 1 --2- 172.21.15.62:0/4012220678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdeb813cc60 0x7fdeb809eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdeb7fff640 10 monclient(hunting): get_auth_request con 0x7fdeb813cc60 auth_method 0 2024-04-19T01:42:25.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdeb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdeb7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdeb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdeb6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdeb6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdeb6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdeb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdeb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdeb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdeb6ffd640 10 monclient(hunting): handle_auth_done global_id 4256 payload 824 2024-04-19T01:42:25.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdeb6ffd640 1 -- 172.21.15.62:0/4012220678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdeb80a60b0 msgr2=0x7fdeb81cfbe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:42:25.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdeb6ffd640 1 --2- 172.21.15.62:0/4012220678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdeb80a60b0 0x7fdeb81cfbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdeb6ffd640 1 -- 172.21.15.62:0/4012220678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdeb813cc60 msgr2=0x7fdeb809eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdeb6ffd640 1 --2- 172.21.15.62:0/4012220678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdeb813cc60 0x7fdeb809eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdeb6ffd640 10 monclient: _finish_hunting 0 2024-04-19T01:42:25.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdeb6ffd640 1 monclient: found mon.noname-c 2024-04-19T01:42:25.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdeb6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:25.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdeb6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7fdeb6ffd640 1 -- 172.21.15.62:0/4012220678 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeb80a0c90 con 0x7fdeb8137780 2024-04-19T01:42:25.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d737d640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:25.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d737d640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-04-19T01:42:25.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d737d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f32d0137780 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:25.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d737d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f32d00a60b0 0x7f32d013c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d737d640 10 monclient(hunting): picked mon.noname-a con 0x7f32d00a60b0 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:25.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d737d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f32d013cc60 0x7f32d013f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d737d640 10 monclient(hunting): picked mon.noname-b con 0x7f32d013cc60 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:25.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d737d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32d009c2e0 con 0x7f32d0137780 2024-04-19T01:42:25.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d737d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d737d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f32d009bd20 con 0x7f32d00a60b0 2024-04-19T01:42:25.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d5119640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f32d0137780 0x7f32d0137b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:33534/0 (socket says 172.21.15.62:33534) 2024-04-19T01:42:25.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d5119640 1 -- 172.21.15.62:0/867145551 learned_addr learned my addr 172.21.15.62:0/867145551 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:42:25.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d4918640 1 --2- 172.21.15.62:0/867145551 >> v2:172.21.15.62:3300/0 conn(0x7f32d00a60b0 0x7f32d013c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d737d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d737d640 1 -- 172.21.15.62:0/867145551 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f32d009c000 con 0x7f32d013cc60 2024-04-19T01:42:25.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d737d640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:25.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d737d640 10 monclient(hunting): authenticate will time out at 931.535394s 2024-04-19T01:42:25.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d4918640 10 monclient(hunting): get_auth_request con 0x7f32d00a60b0 auth_method 0 2024-04-19T01:42:25.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d4918640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d4918640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d4918640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d4918640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d4918640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d4918640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d4918640 10 monclient(hunting): handle_auth_done global_id 4257 payload 274 2024-04-19T01:42:25.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d4918640 1 -- 172.21.15.62:0/867145551 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f32d013cc60 msgr2=0x7f32d013f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:25.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d4918640 1 --2- 172.21.15.62:0/867145551 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f32d013cc60 0x7f32d013f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d4918640 1 -- 172.21.15.62:0/867145551 >> v1:172.21.15.62:6789/0 conn(0x7f32d0137780 legacy=0x7f32d0137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:25.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d4918640 10 monclient: _finish_hunting 0 2024-04-19T01:42:25.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d4918640 1 monclient: found mon.noname-a 2024-04-19T01:42:25.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d4918640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:25.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d4918640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:25.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.710+0000 7f32d4918640 1 -- 172.21.15.62:0/867145551 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32d0138270 con 0x7f32d00a60b0 2024-04-19T01:42:25.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.714+0000 7fdeb6ffd640 1 -- 172.21.15.62:0/4012220678 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdeb8137780 msgr2=0x7fdeb809e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:25.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.714+0000 7fdeb6ffd640 1 --2- 172.21.15.62:0/4012220678 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdeb8137780 0x7fdeb809e760 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdeac01ca40 tx=0x7fdeac018f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.714+0000 7fde9ffff640 1 -- 172.21.15.62:0/4012220678 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeac0b9070 con 0x7fdeb8137780 2024-04-19T01:42:25.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.714+0000 7fde9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.714+0000 7fde9ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:25.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.714+0000 7fde9ffff640 10 monclient: dump: 2024-04-19T01:42:25.733 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.733 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.733 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.733 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.733 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.734 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.734 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.734 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.734 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.734 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.714+0000 7fde9ffff640 10 monclient: _finish_auth 0 2024-04-19T01:42:25.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.714+0000 7fde9ffff640 10 monclient: _check_auth_tickets 2024-04-19T01:42:25.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.714+0000 7fde9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:25.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.714+0000 7fde9ffff640 1 -- 172.21.15.62:0/4012220678 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdeac0b5690 con 0x7fdeb8137780 2024-04-19T01:42:25.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.714+0000 7fde9ffff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:25.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.714+0000 7fde9ffff640 1 -- 172.21.15.62:0/4012220678 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeac0b0040 con 0x7fdeb8137780 2024-04-19T01:42:25.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.714+0000 7fde9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.714+0000 7fde9ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:25.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.714+0000 7fde9ffff640 10 monclient: dump: 2024-04-19T01:42:25.735 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.735 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.735 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.735 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.735 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.735 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.735 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.735 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.735 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.735 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.714+0000 7fdebdeb1640 5 monclient: authenticate success, global_id 4256 2024-04-19T01:42:25.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.714+0000 7fdebdeb1640 10 monclient: _renew_subs 2024-04-19T01:42:25.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.714+0000 7fdebdeb1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.714+0000 7fdebdeb1640 1 -- 172.21.15.62:0/4012220678 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdeb80a3420 con 0x7fdeb8137780 2024-04-19T01:42:25.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.714+0000 7fdeb5ffb640 4 set_mon_vals no callback set 2024-04-19T01:42:25.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.714+0000 7fdebdeb1640 10 monclient: _renew_subs 2024-04-19T01:42:25.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.714+0000 7fdebdeb1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.714+0000 7fdebdeb1640 1 -- 172.21.15.62:0/4012220678 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdeb80a03b0 con 0x7fdeb8137780 2024-04-19T01:42:25.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.714+0000 7fde9dffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:25.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.714+0000 7fde9dffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:25.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.714+0000 7fde9dffb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.714+0000 7fde9dffb640 1 -- 172.21.15.62:0/4012220678 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde90005090 con 0x7fdeb8137780 2024-04-19T01:42:25.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.714+0000 7fde9ffff640 1 -- 172.21.15.62:0/4012220678 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fdeac0bc070 con 0x7fdeb8137780 2024-04-19T01:42:25.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.718+0000 7fde9ffff640 1 --2- 172.21.15.62:0/4012220678 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fdea4036470 0x7fdea4038930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.714+0000 7f32d4918640 1 -- 172.21.15.62:0/867145551 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32d00a60b0 msgr2=0x7f32d013c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:25.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.718+0000 7f32d4918640 1 --2- 172.21.15.62:0/867145551 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32d00a60b0 0x7f32d013c720 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f32c00097c0 tx=0x7f32c001c180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=856d2554eba99dfa server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.718+0000 7fde9ffff640 1 -- 172.21.15.62:0/4012220678 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7fdeac0e3680 con 0x7fdeb8137780 2024-04-19T01:42:25.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.718+0000 7fdeb77fe640 1 --2- 172.21.15.62:0/4012220678 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fdea4036470 0x7fdea4038930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.718+0000 7fdeb77fe640 10 monclient: get_auth_request con 0x7fdea4036470 auth_method 0 2024-04-19T01:42:25.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.718+0000 7fdeb77fe640 1 --2- 172.21.15.62:0/4012220678 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fdea4036470 0x7fdea4038930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdea8003c80 tx=0x7fdea80079c0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.718+0000 7f32c7fff640 1 -- 172.21.15.62:0/867145551 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c00b0070 con 0x7f32d00a60b0 2024-04-19T01:42:25.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.718+0000 7f32c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.718+0000 7f32c7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:25.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.718+0000 7f32c7fff640 10 monclient: dump: 2024-04-19T01:42:25.739 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.739 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.739 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.739 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.740 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.740 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.740 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.740 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.740 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.740 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.718+0000 7f32c7fff640 10 monclient: _finish_auth 0 2024-04-19T01:42:25.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.718+0000 7f32c7fff640 10 monclient: _check_auth_tickets 2024-04-19T01:42:25.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.718+0000 7f32c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:25.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.718+0000 7f32c7fff640 1 -- 172.21.15.62:0/867145551 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32c00ab600 con 0x7f32d00a60b0 2024-04-19T01:42:25.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.718+0000 7f32c7fff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:25.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.718+0000 7f32c7fff640 1 -- 172.21.15.62:0/867145551 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c00ab9d0 con 0x7f32d00a60b0 2024-04-19T01:42:25.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.718+0000 7f32c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.718+0000 7f32c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:25.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.718+0000 7f32c7fff640 10 monclient: dump: 2024-04-19T01:42:25.741 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.741 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.741 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.741 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.741 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.741 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.741 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.741 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.741 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.741 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.718+0000 7f32d737d640 5 monclient: authenticate success, global_id 4257 2024-04-19T01:42:25.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.718+0000 7f32d737d640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:25.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.718+0000 7f32d737d640 4 set_mon_vals no callback set 2024-04-19T01:42:25.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.722+0000 7f32d737d640 10 monclient: shutdown 2024-04-19T01:42:25.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.722+0000 7f32d737d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:25.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.722+0000 7f32d737d640 1 -- 172.21.15.62:0/867145551 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32d00a60b0 msgr2=0x7f32d013c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.722+0000 7f32d737d640 1 --2- 172.21.15.62:0/867145551 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32d00a60b0 0x7f32d013c720 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f32c00097c0 tx=0x7f32c001c180 comp rx=0 tx=0).stop 2024-04-19T01:42:25.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.722+0000 7fde9ffff640 1 -- 172.21.15.62:0/4012220678 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fdeb809c2e0 con 0x7fdeb8137780 2024-04-19T01:42:25.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.722+0000 7fde9ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:25.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.722+0000 7fde9ffff640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:25.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.722+0000 7f32d737d640 1 -- 172.21.15.62:0/867145551 shutdown_connections 2024-04-19T01:42:25.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.722+0000 7f32d737d640 1 --2- 172.21.15.62:0/867145551 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f32d013cc60 0x7f32d013f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.722+0000 7f32d737d640 1 --2- 172.21.15.62:0/867145551 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32d00a60b0 0x7f32d013c720 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.722+0000 7f32d737d640 1 -- 172.21.15.62:0/867145551 >> 172.21.15.62:0/867145551 conn(0x7f32d00aa1d0 msgr2=0x7f32d00a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:25.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d737d640 1 -- 172.21.15.62:0/867145551 shutdown_connections 2024-04-19T01:42:25.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d737d640 1 -- 172.21.15.62:0/867145551 wait complete. 2024-04-19T01:42:25.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d737d640 10 monclient: build_initial_monmap 2024-04-19T01:42:25.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d737d640 10 monclient: monmap: 2024-04-19T01:42:25.748 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:25.748 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:25.748 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:25.748 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:25.748 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:25.748 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.748 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:25.749 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:25.749 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:25.749 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d737d640 1 Processor -- start 2024-04-19T01:42:25.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d737d640 1 -- start start 2024-04-19T01:42:25.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d737d640 10 monclient: init 2024-04-19T01:42:25.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d737d640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:25.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d737d640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-19T01:42:25.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d737d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f32d00a60b0 0x7f32d01cf5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d737d640 10 monclient(hunting): picked mon.noname-c con 0x7f32d00a60b0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:25.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d737d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f32d0137780 0x7f32d0136480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d737d640 10 monclient(hunting): picked mon.noname-b con 0x7f32d0137780 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:25.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d5119640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f32d00a60b0 0x7f32d01cf5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d737d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32d013cc60 0x7f32d009fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d737d640 10 monclient(hunting): picked mon.noname-a con 0x7f32d013cc60 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:25.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d5119640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f32d00a60b0 0x7f32d01cf5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60360/0 (socket says 172.21.15.62:60360) 2024-04-19T01:42:25.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d5119640 1 -- 172.21.15.62:0/4170700954 learned_addr learned my addr 172.21.15.62:0/4170700954 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:25.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d737d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d737d640 1 -- 172.21.15.62:0/4170700954 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f32d0142140 con 0x7f32d013cc60 2024-04-19T01:42:25.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d737d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d737d640 1 -- 172.21.15.62:0/4170700954 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f32d0141fc0 con 0x7f32d0137780 2024-04-19T01:42:25.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d737d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d737d640 1 -- 172.21.15.62:0/4170700954 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f32d01422c0 con 0x7f32d00a60b0 2024-04-19T01:42:25.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d737d640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:25.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d591a640 1 --2- 172.21.15.62:0/4170700954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32d013cc60 0x7f32d009fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d737d640 10 monclient(hunting): authenticate will time out at 931.551287s 2024-04-19T01:42:25.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d5119640 10 monclient(hunting): get_auth_request con 0x7f32d00a60b0 auth_method 0 2024-04-19T01:42:25.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d5119640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d5119640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d5119640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d591a640 10 monclient(hunting): get_auth_request con 0x7f32d013cc60 auth_method 0 2024-04-19T01:42:25.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d591a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d591a640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d591a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d5119640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d5119640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d4918640 1 --2- 172.21.15.62:0/4170700954 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f32d0137780 0x7f32d0136480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d5119640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d591a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d591a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d591a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d4918640 10 monclient(hunting): get_auth_request con 0x7f32d0137780 auth_method 0 2024-04-19T01:42:25.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d4918640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d4918640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d4918640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d5119640 10 monclient(hunting): handle_auth_done global_id 4259 payload 824 2024-04-19T01:42:25.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d5119640 1 -- 172.21.15.62:0/4170700954 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f32d0137780 msgr2=0x7f32d0136480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d5119640 1 --2- 172.21.15.62:0/4170700954 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f32d0137780 0x7f32d0136480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d5119640 1 -- 172.21.15.62:0/4170700954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32d013cc60 msgr2=0x7f32d009fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d5119640 1 --2- 172.21.15.62:0/4170700954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32d013cc60 0x7f32d009fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d5119640 10 monclient: _finish_hunting 0 2024-04-19T01:42:25.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d5119640 1 monclient: found mon.noname-c 2024-04-19T01:42:25.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d5119640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:25.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d5119640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d5119640 1 -- 172.21.15.62:0/4170700954 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32d00a2220 con 0x7f32d00a60b0 2024-04-19T01:42:25.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.726+0000 7f32d591a640 1 --2- 172.21.15.62:0/4170700954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32d013cc60 0x7f32d009fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:42:25.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.730+0000 7f32d5119640 1 -- 172.21.15.62:0/4170700954 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f32d00a60b0 msgr2=0x7f32d01cf5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:25.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.730+0000 7f32d5119640 1 --2- 172.21.15.62:0/4170700954 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f32d00a60b0 0x7f32d01cf5c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f32cc00c4a0 tx=0x7f32cc00c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.730+0000 7f32c5ffb640 1 -- 172.21.15.62:0/4170700954 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32cc017070 con 0x7f32d00a60b0 2024-04-19T01:42:25.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.730+0000 7f32c5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.730+0000 7f32c5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:25.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.730+0000 7f32c5ffb640 10 monclient: dump: 2024-04-19T01:42:25.754 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.754 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.754 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.754 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.754 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.754 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.754 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.754 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.754 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.754 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.730+0000 7f32c5ffb640 10 monclient: _finish_auth 0 2024-04-19T01:42:25.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.730+0000 7f32c5ffb640 10 monclient: _check_auth_tickets 2024-04-19T01:42:25.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.730+0000 7f32c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:25.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.730+0000 7f32c5ffb640 1 -- 172.21.15.62:0/4170700954 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32cc007070 con 0x7f32d00a60b0 2024-04-19T01:42:25.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.730+0000 7f32c5ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:25.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.730+0000 7f32c5ffb640 1 -- 172.21.15.62:0/4170700954 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32cc007460 con 0x7f32d00a60b0 2024-04-19T01:42:25.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.730+0000 7f32c5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.730+0000 7f32c5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:25.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.730+0000 7f32c5ffb640 10 monclient: dump: 2024-04-19T01:42:25.755 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.755 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.755 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.755 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.755 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.755 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.755 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.755 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.756 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.756 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.730+0000 7f32c77fe640 4 set_mon_vals no callback set 2024-04-19T01:42:25.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.730+0000 7f32d737d640 5 monclient: authenticate success, global_id 4259 2024-04-19T01:42:25.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.730+0000 7f32d737d640 10 monclient: _renew_subs 2024-04-19T01:42:25.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.730+0000 7f32d737d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.730+0000 7f32d737d640 1 -- 172.21.15.62:0/4170700954 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32d00a3420 con 0x7f32d00a60b0 2024-04-19T01:42:25.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.730+0000 7f32d737d640 10 monclient: _renew_subs 2024-04-19T01:42:25.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.730+0000 7f32d737d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.730+0000 7f32d737d640 1 -- 172.21.15.62:0/4170700954 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f32d0120950 con 0x7f32d00a60b0 2024-04-19T01:42:25.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.730+0000 7f32c5ffb640 1 -- 172.21.15.62:0/4170700954 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f32cc01a070 con 0x7f32d00a60b0 2024-04-19T01:42:25.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.730+0000 7f32c5ffb640 1 --2- 172.21.15.62:0/4170700954 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f32b40364a0 0x7f32b4038960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.730+0000 7f32c5ffb640 1 -- 172.21.15.62:0/4170700954 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f32cc048790 con 0x7f32d00a60b0 2024-04-19T01:42:25.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.734+0000 7f32d737d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:25.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.734+0000 7f32d737d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:25.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.734+0000 7f32d4918640 1 --2- 172.21.15.62:0/4170700954 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f32b40364a0 0x7f32b4038960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.734+0000 7f32d737d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.734+0000 7f32d737d640 1 -- 172.21.15.62:0/4170700954 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32d0068ab0 con 0x7f32d00a60b0 2024-04-19T01:42:25.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.734+0000 7f32d4918640 10 monclient: get_auth_request con 0x7f32b40364a0 auth_method 0 2024-04-19T01:42:25.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.734+0000 7f32d4918640 1 --2- 172.21.15.62:0/4170700954 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f32b40364a0 0x7f32b4038960 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f32c00ab000 tx=0x7f32c0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.742+0000 7f32c5ffb640 1 -- 172.21.15.62:0/4170700954 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f32cc010300 con 0x7f32d00a60b0 2024-04-19T01:42:25.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.742+0000 7f32c5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:25.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.742+0000 7f32c5ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:25.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f556893a640 10 monclient: get_monmap_and_config 2024-04-19T01:42:25.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f556893a640 10 monclient: build_initial_monmap 2024-04-19T01:42:25.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f556893a640 10 monclient: monmap: 2024-04-19T01:42:25.811 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:25.811 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.811 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:25.815078+0000 2024-04-19T01:42:25.811 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:25.815078+0000 2024-04-19T01:42:25.811 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:25.811 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.811 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:25.812 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:25.812 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:25.812 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:25.812 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:25.812 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f556893a640 1 Processor -- start 2024-04-19T01:42:25.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f556893a640 1 -- start start 2024-04-19T01:42:25.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f556893a640 10 monclient: init 2024-04-19T01:42:25.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f556893a640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:25.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f556893a640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-04-19T01:42:25.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f556893a640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f5564137780 0x7f5564137b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f556893a640 10 monclient(hunting): picked mon.noname-c con 0x7f5564137780 addr v2:172.21.15.71:3300/0 2024-04-19T01:42:25.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f556893a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f55641380a0 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:25.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f556893a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55640a60b0 0x7f55640a4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f556893a640 10 monclient(hunting): picked mon.noname-b con 0x7f55640a60b0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:25.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f556893a640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f556409c2e0 con 0x7f55641380a0 2024-04-19T01:42:25.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f556893a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f556893a640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f556409bd20 con 0x7f5564137780 2024-04-19T01:42:25.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f556893a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f556893a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f556409c000 con 0x7f55640a60b0 2024-04-19T01:42:25.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f556893a640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:25.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f556893a640 10 monclient(hunting): authenticate will time out at 931.634344s 2024-04-19T01:42:25.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f5562d9d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55640a60b0 0x7f55640a4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f5562d9d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55640a60b0 0x7f55640a4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60366/0 (socket says 172.21.15.62:60366) 2024-04-19T01:42:25.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f5562d9d640 1 -- 172.21.15.62:0/835936479 learned_addr learned my addr 172.21.15.62:0/835936479 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:25.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f5562d9d640 10 monclient(hunting): get_auth_request con 0x7f55640a60b0 auth_method 0 2024-04-19T01:42:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f5562d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f5562d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f5562d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f5561d9b640 1 --1- 172.21.15.62:0/835936479 >> v1:172.21.15.62:6789/0 conn(0x7f55641380a0 0x7f55640a45d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:33550/0 (socket says 172.21.15.62:33550) 2024-04-19T01:42:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f5562d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f5562d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f5562d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f556259c640 1 --2- 172.21.15.62:0/835936479 >> v2:172.21.15.71:3300/0 conn(0x7f5564137780 0x7f5564137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f556259c640 10 monclient(hunting): get_auth_request con 0x7f5564137780 auth_method 0 2024-04-19T01:42:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f556259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f556259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f556259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f5562d9d640 10 monclient(hunting): handle_auth_done global_id 4262 payload 274 2024-04-19T01:42:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f5562d9d640 1 -- 172.21.15.62:0/835936479 >> v2:172.21.15.71:3300/0 conn(0x7f5564137780 msgr2=0x7f5564137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f5562d9d640 1 --2- 172.21.15.62:0/835936479 >> v2:172.21.15.71:3300/0 conn(0x7f5564137780 0x7f5564137b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f5562d9d640 1 -- 172.21.15.62:0/835936479 >> v1:172.21.15.62:6789/0 conn(0x7f55641380a0 legacy=0x7f55640a45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:25.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f5562d9d640 10 monclient: _finish_hunting 0 2024-04-19T01:42:25.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f5562d9d640 1 monclient: found mon.noname-b 2024-04-19T01:42:25.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f5562d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:25.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f5562d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.810+0000 7f5562d9d640 1 -- 172.21.15.62:0/835936479 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5564073210 con 0x7f55640a60b0 2024-04-19T01:42:25.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.814+0000 7f5562d9d640 1 -- 172.21.15.62:0/835936479 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55640a60b0 msgr2=0x7f55640a4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:25.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.814+0000 7f5562d9d640 1 --2- 172.21.15.62:0/835936479 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55640a60b0 0x7f55640a4ce0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f555c009e50 tx=0x7f555c01c710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0b62dcf6e1d80ff server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.814+0000 7f556159a640 1 -- 172.21.15.62:0/835936479 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555c0ac020 con 0x7f55640a60b0 2024-04-19T01:42:25.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.814+0000 7f556159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.814+0000 7f556159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:42:25.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.814+0000 7f556159a640 10 monclient: dump: 2024-04-19T01:42:25.819 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.819 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.819 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.819 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.819 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.819 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.819 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.819 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.819 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.819 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.814+0000 7f556159a640 10 monclient: _finish_auth 0 2024-04-19T01:42:25.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.814+0000 7f556159a640 10 monclient: _check_auth_tickets 2024-04-19T01:42:25.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.814+0000 7f556159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:25.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.814+0000 7f556159a640 1 -- 172.21.15.62:0/835936479 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f555c0a6470 con 0x7f55640a60b0 2024-04-19T01:42:25.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.814+0000 7f556159a640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:25.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.814+0000 7f556159a640 1 -- 172.21.15.62:0/835936479 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555c0a6840 con 0x7f55640a60b0 2024-04-19T01:42:25.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.814+0000 7f556159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.814+0000 7f556159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:25.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.814+0000 7f556159a640 10 monclient: dump: 2024-04-19T01:42:25.820 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.821 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.821 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.821 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.821 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.821 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.821 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.821 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.821 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.821 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.814+0000 7f556893a640 5 monclient: authenticate success, global_id 4262 2024-04-19T01:42:25.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.814+0000 7f556893a640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:25.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.814+0000 7f556893a640 4 set_mon_vals no callback set 2024-04-19T01:42:25.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f556893a640 10 monclient: shutdown 2024-04-19T01:42:25.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f556893a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:25.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f556893a640 1 -- 172.21.15.62:0/835936479 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55640a60b0 msgr2=0x7f55640a4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f556893a640 1 --2- 172.21.15.62:0/835936479 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55640a60b0 0x7f55640a4ce0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f555c009e50 tx=0x7f555c01c710 comp rx=0 tx=0).stop 2024-04-19T01:42:25.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f556893a640 1 -- 172.21.15.62:0/835936479 shutdown_connections 2024-04-19T01:42:25.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f556893a640 1 --2- 172.21.15.62:0/835936479 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55640a60b0 0x7f55640a4ce0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f556893a640 1 --2- 172.21.15.62:0/835936479 >> v2:172.21.15.71:3300/0 conn(0x7f5564137780 0x7f5564137b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f556893a640 1 -- 172.21.15.62:0/835936479 >> 172.21.15.62:0/835936479 conn(0x7f55640aa1d0 msgr2=0x7f55640a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:25.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f556893a640 1 -- 172.21.15.62:0/835936479 shutdown_connections 2024-04-19T01:42:25.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f556893a640 1 -- 172.21.15.62:0/835936479 wait complete. 2024-04-19T01:42:25.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f556893a640 10 monclient: build_initial_monmap 2024-04-19T01:42:25.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f556893a640 10 monclient: monmap: 2024-04-19T01:42:25.822 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:25.822 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:25.822 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:25.822 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:25.823 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:25.823 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.823 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:25.823 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:25.823 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:25.823 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f556893a640 1 Processor -- start 2024-04-19T01:42:25.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f556893a640 1 -- start start 2024-04-19T01:42:25.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f556893a640 10 monclient: init 2024-04-19T01:42:25.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f556893a640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:25.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f556893a640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-19T01:42:25.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f556893a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55640a60b0 0x7f55641cd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f556893a640 10 monclient(hunting): picked mon.noname-b con 0x7f55640a60b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:25.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f556893a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5564137780 0x7f556413e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f556893a640 10 monclient(hunting): picked mon.noname-a con 0x7f5564137780 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:25.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f556893a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55641380a0 0x7f55641363f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f556893a640 10 monclient(hunting): picked mon.noname-c con 0x7f55641380a0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:25.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f556893a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f556893a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5564142170 con 0x7f5564137780 2024-04-19T01:42:25.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f556893a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f556893a640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f55640a5430 con 0x7f55640a60b0 2024-04-19T01:42:25.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f556893a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:25.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f556893a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f55641422f0 con 0x7f55641380a0 2024-04-19T01:42:25.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f556893a640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:25.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f556893a640 10 monclient(hunting): authenticate will time out at 931.641771s 2024-04-19T01:42:25.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f5562d9d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55641380a0 0x7f55641363f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f5562d9d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55641380a0 0x7f55641363f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60382/0 (socket says 172.21.15.62:60382) 2024-04-19T01:42:25.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f5562d9d640 1 -- 172.21.15.62:0/33811486 learned_addr learned my addr 172.21.15.62:0/33811486 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:25.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f5562d9d640 10 monclient(hunting): get_auth_request con 0x7f55641380a0 auth_method 0 2024-04-19T01:42:25.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f5562d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f5562d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f5562d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f5562d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:25.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f5562d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:25.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f5562d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:25.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f5561d9b640 1 --2- 172.21.15.62:0/33811486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5564137780 0x7f556413e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f5561d9b640 10 monclient(hunting): get_auth_request con 0x7f5564137780 auth_method 0 2024-04-19T01:42:25.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f5561d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:25.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f5561d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:25.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f5561d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:25.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f5562d9d640 10 monclient(hunting): handle_auth_done global_id 4265 payload 824 2024-04-19T01:42:25.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f5562d9d640 1 -- 172.21.15.62:0/33811486 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55640a60b0 msgr2=0x7f55641cd510 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-19T01:42:25.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f5562d9d640 1 --2- 172.21.15.62:0/33811486 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55640a60b0 0x7f55641cd510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f5562d9d640 1 -- 172.21.15.62:0/33811486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5564137780 msgr2=0x7f556413e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f5562d9d640 1 --2- 172.21.15.62:0/33811486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5564137780 0x7f556413e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f5562d9d640 10 monclient: _finish_hunting 0 2024-04-19T01:42:25.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f5562d9d640 1 monclient: found mon.noname-c 2024-04-19T01:42:25.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f5562d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:25.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f5562d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f5562d9d640 1 -- 172.21.15.62:0/33811486 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5564120220 con 0x7f55641380a0 2024-04-19T01:42:25.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.818+0000 7f5561d9b640 1 --2- 172.21.15.62:0/33811486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5564137780 0x7f556413e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T01:42:25.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.822+0000 7f5562d9d640 1 -- 172.21.15.62:0/33811486 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55641380a0 msgr2=0x7f55641363f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:25.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.822+0000 7f5562d9d640 1 --2- 172.21.15.62:0/33811486 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55641380a0 0x7f55641363f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f555c01cfb0 tx=0x7f555c0194b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.822+0000 7f5552ffd640 1 -- 172.21.15.62:0/33811486 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555c0ac040 con 0x7f55641380a0 2024-04-19T01:42:25.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.822+0000 7f5552ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.822+0000 7f5552ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:25.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.822+0000 7f5552ffd640 10 monclient: dump: 2024-04-19T01:42:25.830 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.830 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.830 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.830 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.830 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.830 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.830 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.830 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.830 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.830 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.822+0000 7f5552ffd640 10 monclient: _finish_auth 0 2024-04-19T01:42:25.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.822+0000 7f5552ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:42:25.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.822+0000 7f5552ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:25.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.822+0000 7f5552ffd640 1 -- 172.21.15.62:0/33811486 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f555c0a7650 con 0x7f55641380a0 2024-04-19T01:42:25.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.822+0000 7f556893a640 5 monclient: authenticate success, global_id 4265 2024-04-19T01:42:25.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.822+0000 7f556893a640 10 monclient: _renew_subs 2024-04-19T01:42:25.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.822+0000 7f556893a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.822+0000 7f556893a640 1 -- 172.21.15.62:0/33811486 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55641d0410 con 0x7f55641380a0 2024-04-19T01:42:25.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.822+0000 7f5552ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:25.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.822+0000 7f5552ffd640 1 -- 172.21.15.62:0/33811486 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555c0a7a20 con 0x7f55641380a0 2024-04-19T01:42:25.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.822+0000 7f5552ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:25.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.822+0000 7f5552ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:25.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.822+0000 7f5552ffd640 10 monclient: dump: 2024-04-19T01:42:25.831 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:25.831 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:25.831 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.831 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:25.832 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:25.832 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:25.832 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:25.832 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:25.832 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:25.832 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:25.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.822+0000 7f556893a640 10 monclient: _renew_subs 2024-04-19T01:42:25.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.822+0000 7f556893a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.822+0000 7f556893a640 1 -- 172.21.15.62:0/33811486 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f55640a2400 con 0x7f55641380a0 2024-04-19T01:42:25.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.822+0000 7f5560d99640 4 set_mon_vals no callback set 2024-04-19T01:42:25.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.822+0000 7f556893a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:25.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.822+0000 7f556893a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:25.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.822+0000 7f556893a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.822+0000 7f556893a640 1 -- 172.21.15.62:0/33811486 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5564068ab0 con 0x7f55641380a0 2024-04-19T01:42:25.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.822+0000 7f5552ffd640 1 -- 172.21.15.62:0/33811486 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f555c0af020 con 0x7f55641380a0 2024-04-19T01:42:25.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.822+0000 7f5552ffd640 1 --2- 172.21.15.62:0/33811486 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f55440364a0 0x7f5544038960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:25.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.822+0000 7f5552ffd640 1 -- 172.21.15.62:0/33811486 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f555c0e3630 con 0x7f55641380a0 2024-04-19T01:42:25.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.822+0000 7f556259c640 1 --2- 172.21.15.62:0/33811486 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f55440364a0 0x7f5544038960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:25.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.822+0000 7f556259c640 10 monclient: get_auth_request con 0x7f55440364a0 auth_method 0 2024-04-19T01:42:25.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.826+0000 7f556259c640 1 --2- 172.21.15.62:0/33811486 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f55440364a0 0x7f5544038960 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f555400b810 tx=0x7f5554008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:25.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.830+0000 7f5552ffd640 1 -- 172.21.15.62:0/33811486 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f555c0c1060 con 0x7f55641380a0 2024-04-19T01:42:25.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.830+0000 7f5552ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:25.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.830+0000 7f5552ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:25.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.894+0000 7fde9dffb640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-19T01:42:25.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.894+0000 7fde9dffb640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-19T01:42:25.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.894+0000 7fde9dffb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.894+0000 7fde9dffb640 1 -- 172.21.15.62:0/4012220678 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fde90005380 con 0x7fdeb8137780 2024-04-19T01:42:25.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.894+0000 7fde9ffff640 1 -- 172.21.15.62:0/4012220678 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fdeac0a97e0 con 0x7fdeb8137780 2024-04-19T01:42:25.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.894+0000 7fde9ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-19T01:42:25.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.894+0000 7fde9ffff640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:25.895 INFO:teuthology.orchestra.run.smithi062.stdout:47244640257 2024-04-19T01:42:25.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.894+0000 7fde9dffb640 1 -- 172.21.15.62:0/4012220678 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fdea4036470 msgr2=0x7fdea4038930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.894+0000 7fde9dffb640 1 --2- 172.21.15.62:0/4012220678 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fdea4036470 0x7fdea4038930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdea8003c80 tx=0x7fdea80079c0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.894+0000 7fde9dffb640 10 monclient: shutdown 2024-04-19T01:42:25.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.894+0000 7fde9dffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:25.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.894+0000 7fde9dffb640 1 -- 172.21.15.62:0/4012220678 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdeb8137780 msgr2=0x7fdeb809e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.894+0000 7fde9dffb640 1 --2- 172.21.15.62:0/4012220678 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdeb8137780 0x7fdeb809e760 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdeac01ca40 tx=0x7fdeac018f40 comp rx=0 tx=0).stop 2024-04-19T01:42:25.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.894+0000 7fde9dffb640 1 -- 172.21.15.62:0/4012220678 shutdown_connections 2024-04-19T01:42:25.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.894+0000 7fde9dffb640 1 --2- 172.21.15.62:0/4012220678 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fdea4036470 0x7fdea4038930 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.894+0000 7fde9dffb640 1 --2- 172.21.15.62:0/4012220678 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdeb8137780 0x7fdeb809e760 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.894+0000 7fde9dffb640 1 --2- 172.21.15.62:0/4012220678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdeb813cc60 0x7fdeb809eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.894+0000 7fde9dffb640 1 --2- 172.21.15.62:0/4012220678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdeb80a60b0 0x7fdeb81cfbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.894+0000 7fde9dffb640 1 -- 172.21.15.62:0/4012220678 >> 172.21.15.62:0/4012220678 conn(0x7fdeb80aa1d0 msgr2=0x7fdeb8137b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:25.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.894+0000 7fde9dffb640 1 -- 172.21.15.62:0/4012220678 shutdown_connections 2024-04-19T01:42:25.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.894+0000 7fde9dffb640 1 -- 172.21.15.62:0/4012220678 wait complete. 2024-04-19T01:42:25.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.906+0000 7f0bea69c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-19T01:42:25.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.906+0000 7f0bea69c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-19T01:42:25.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.906+0000 7f0bea69c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:25.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.906+0000 7f0bea69c640 1 -- 172.21.15.62:0/3788694605 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f0be4068730 con 0x7f0be40a2970 2024-04-19T01:42:25.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.906+0000 7f0be0ff9640 1 -- 172.21.15.62:0/3788694605 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0bd80b1020 con 0x7f0be40a2970 2024-04-19T01:42:25.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.906+0000 7f0be0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-19T01:42:25.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.906+0000 7f0be0ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:25.909 INFO:teuthology.orchestra.run.smithi062.stdout:51539607553 2024-04-19T01:42:25.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.910+0000 7f0bea69c640 1 -- 172.21.15.62:0/3788694605 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0bd0036470 msgr2=0x7f0bd0038930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.910+0000 7f0bea69c640 1 --2- 172.21.15.62:0/3788694605 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0bd0036470 0x7f0bd0038930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0bd4009c20 tx=0x7f0bd4008040 comp rx=0 tx=0).stop 2024-04-19T01:42:25.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.910+0000 7f0bea69c640 10 monclient: shutdown 2024-04-19T01:42:25.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.910+0000 7f0bea69c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:25.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.910+0000 7f0bea69c640 1 -- 172.21.15.62:0/3788694605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0be40a2970 msgr2=0x7f0be412ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.910+0000 7f0bea69c640 1 --2- 172.21.15.62:0/3788694605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0be40a2970 0x7f0be412ea50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0bd80b2000 tx=0x7f0bd801f030 comp rx=0 tx=0).stop 2024-04-19T01:42:25.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.910+0000 7f0bea69c640 1 -- 172.21.15.62:0/3788694605 shutdown_connections 2024-04-19T01:42:25.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.910+0000 7f0bea69c640 1 --2- 172.21.15.62:0/3788694605 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0be41355e0 0x7f0be409ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.910+0000 7f0bea69c640 1 --2- 172.21.15.62:0/3788694605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0be40a2970 0x7f0be412ea50 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.910+0000 7f0bea69c640 1 --2- 172.21.15.62:0/3788694605 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0bd0036470 0x7f0bd0038930 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.910+0000 7f0bea69c640 1 --2- 172.21.15.62:0/3788694605 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0be40726e0 0x7f0be412e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.910+0000 7f0bea69c640 1 -- 172.21.15.62:0/3788694605 >> 172.21.15.62:0/3788694605 conn(0x7f0be40a8030 msgr2=0x7f0be4134a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:25.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.910+0000 7f0bea69c640 1 -- 172.21.15.62:0/3788694605 shutdown_connections 2024-04-19T01:42:25.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.910+0000 7f0bea69c640 1 -- 172.21.15.62:0/3788694605 wait complete. 2024-04-19T01:42:25.912 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640257 for osd.5 2024-04-19T01:42:25.925 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607554 got 51539607553 for osd.4 2024-04-19T01:42:25.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.926+0000 7f973688a640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-19T01:42:25.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.926+0000 7f973688a640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-19T01:42:25.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.926+0000 7f973688a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:25.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.926+0000 7f973688a640 1 -- 172.21.15.62:0/2304285399 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f9730068730 con 0x7f9730072b00 2024-04-19T01:42:25.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.926+0000 7f972cff9640 1 -- 172.21.15.62:0/2304285399 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f97240476c0 con 0x7f9730072b00 2024-04-19T01:42:25.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.926+0000 7f972cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-19T01:42:25.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.926+0000 7f972cff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:25.928 INFO:teuthology.orchestra.run.smithi062.stdout:47244640258 2024-04-19T01:42:25.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.930+0000 7f973688a640 1 -- 172.21.15.62:0/2304285399 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f971c036420 msgr2=0x7f971c0388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.930+0000 7f973688a640 1 --2- 172.21.15.62:0/2304285399 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f971c036420 0x7f971c0388e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9720007fd0 tx=0x7f9720007e40 comp rx=0 tx=0).stop 2024-04-19T01:42:25.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.930+0000 7f973688a640 10 monclient: shutdown 2024-04-19T01:42:25.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.930+0000 7f973688a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:25.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.930+0000 7f973688a640 1 -- 172.21.15.62:0/2304285399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9730072b00 msgr2=0x7f973013b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:25.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.930+0000 7f973688a640 1 --2- 172.21.15.62:0/2304285399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9730072b00 0x7f973013b430 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f972400c470 tx=0x7f972400c930 comp rx=0 tx=0).stop 2024-04-19T01:42:25.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.930+0000 7f973688a640 1 -- 172.21.15.62:0/2304285399 shutdown_connections 2024-04-19T01:42:25.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.930+0000 7f973688a640 1 --2- 172.21.15.62:0/2304285399 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97301355e0 0x7f97301341e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.930+0000 7f973688a640 1 --2- 172.21.15.62:0/2304285399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9730072b00 0x7f973013b430 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.930+0000 7f973688a640 1 --2- 172.21.15.62:0/2304285399 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f971c036420 0x7f971c0388e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.930+0000 7f973688a640 1 --2- 172.21.15.62:0/2304285399 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97300726e0 0x7f973011e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:25.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.930+0000 7f973688a640 1 -- 172.21.15.62:0/2304285399 >> 172.21.15.62:0/2304285399 conn(0x7f97300a8030 msgr2=0x7f97300a04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:25.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.930+0000 7f973688a640 1 -- 172.21.15.62:0/2304285399 shutdown_connections 2024-04-19T01:42:25.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.930+0000 7f973688a640 1 -- 172.21.15.62:0/2304285399 wait complete. 2024-04-19T01:42:25.945 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640258 for osd.0 2024-04-19T01:42:25.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.994+0000 7f32d737d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-19T01:42:25.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.994+0000 7f32d737d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-19T01:42:25.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.994+0000 7f32d737d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:25.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.994+0000 7f32d737d640 1 -- 172.21.15.62:0/4170700954 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f32d00a1fa0 con 0x7f32d00a60b0 2024-04-19T01:42:25.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.994+0000 7f32c5ffb640 1 -- 172.21.15.62:0/4170700954 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f32cc0483a0 con 0x7f32d00a60b0 2024-04-19T01:42:25.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.998+0000 7f32c5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-19T01:42:25.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.998+0000 7f32c5ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:25.998 INFO:teuthology.orchestra.run.smithi062.stdout:47244640258 2024-04-19T01:42:26.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.998+0000 7f32d737d640 1 -- 172.21.15.62:0/4170700954 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f32b40364a0 msgr2=0x7f32b4038960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:26.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.998+0000 7f32d737d640 1 --2- 172.21.15.62:0/4170700954 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f32b40364a0 0x7f32b4038960 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f32c00ab000 tx=0x7f32c0008040 comp rx=0 tx=0).stop 2024-04-19T01:42:26.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.998+0000 7f32d737d640 10 monclient: shutdown 2024-04-19T01:42:26.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.998+0000 7f32d737d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:26.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.998+0000 7f32d737d640 1 -- 172.21.15.62:0/4170700954 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f32d00a60b0 msgr2=0x7f32d01cf5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:26.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.998+0000 7f32d737d640 1 --2- 172.21.15.62:0/4170700954 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f32d00a60b0 0x7f32d01cf5c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f32cc00c4a0 tx=0x7f32cc00c960 comp rx=0 tx=0).stop 2024-04-19T01:42:26.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.998+0000 7f32d737d640 1 -- 172.21.15.62:0/4170700954 shutdown_connections 2024-04-19T01:42:26.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.998+0000 7f32d737d640 1 --2- 172.21.15.62:0/4170700954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32d013cc60 0x7f32d009fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:26.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.998+0000 7f32d737d640 1 --2- 172.21.15.62:0/4170700954 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f32d0137780 0x7f32d0136480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:26.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.998+0000 7f32d737d640 1 --2- 172.21.15.62:0/4170700954 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f32b40364a0 0x7f32b4038960 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:26.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.998+0000 7f32d737d640 1 --2- 172.21.15.62:0/4170700954 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f32d00a60b0 0x7f32d01cf5c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:26.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.998+0000 7f32d737d640 1 -- 172.21.15.62:0/4170700954 >> 172.21.15.62:0/4170700954 conn(0x7f32d00aa1d0 msgr2=0x7f32d00a9480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:26.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.998+0000 7f32d737d640 1 -- 172.21.15.62:0/4170700954 shutdown_connections 2024-04-19T01:42:26.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:25.998+0000 7f32d737d640 1 -- 172.21.15.62:0/4170700954 wait complete. 2024-04-19T01:42:26.014 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.3 2024-04-19T01:42:26.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.042+0000 7fe7b97f1640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-19T01:42:26.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.042+0000 7fe7b97f1640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-19T01:42:26.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.042+0000 7fe7b97f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:26.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.042+0000 7fe7b97f1640 1 -- 172.21.15.62:0/2034685977 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fe7b4068710 con 0x7fe7b41380a0 2024-04-19T01:42:26.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.042+0000 7fe79b7fe640 1 -- 172.21.15.62:0/2034685977 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe7b413ddd0 con 0x7fe7b41380a0 2024-04-19T01:42:26.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.042+0000 7fe79b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-19T01:42:26.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.042+0000 7fe79b7fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:26.044 INFO:teuthology.orchestra.run.smithi062.stdout:47244640258 2024-04-19T01:42:26.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.046+0000 7fe7b97f1640 1 -- 172.21.15.62:0/2034685977 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fe7a0036470 msgr2=0x7fe7a0038930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:26.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.046+0000 7fe7b97f1640 1 --2- 172.21.15.62:0/2034685977 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fe7a0036470 0x7fe7a0038930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4006cd0 tx=0x7fe7a4009240 comp rx=0 tx=0).stop 2024-04-19T01:42:26.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.046+0000 7fe7b97f1640 10 monclient: shutdown 2024-04-19T01:42:26.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.046+0000 7fe7b97f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:26.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.046+0000 7fe7b97f1640 1 -- 172.21.15.62:0/2034685977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7b41380a0 msgr2=0x7fe7b40a0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:26.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.046+0000 7fe7b97f1640 1 --2- 172.21.15.62:0/2034685977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7b41380a0 0x7fe7b40a0850 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac01aea0 tx=0x7fe7ac019590 comp rx=0 tx=0).stop 2024-04-19T01:42:26.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.046+0000 7fe7b97f1640 1 -- 172.21.15.62:0/2034685977 shutdown_connections 2024-04-19T01:42:26.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.046+0000 7fe7b97f1640 1 --2- 172.21.15.62:0/2034685977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7b41380a0 0x7fe7b40a0850 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:26.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.046+0000 7fe7b97f1640 1 --2- 172.21.15.62:0/2034685977 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe7b4137780 0x7fe7b4136450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:26.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.046+0000 7fe7b97f1640 1 --2- 172.21.15.62:0/2034685977 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fe7a0036470 0x7fe7a0038930 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:26.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.046+0000 7fe7b97f1640 1 --2- 172.21.15.62:0/2034685977 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe7b40a60b0 0x7fe7b41cf400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:26.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.046+0000 7fe7b97f1640 1 -- 172.21.15.62:0/2034685977 >> 172.21.15.62:0/2034685977 conn(0x7fe7b40aa1d0 msgr2=0x7fe7b413b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:26.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.046+0000 7fe7b97f1640 1 -- 172.21.15.62:0/2034685977 shutdown_connections 2024-04-19T01:42:26.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.046+0000 7fe7b97f1640 1 -- 172.21.15.62:0/2034685977 wait complete. 2024-04-19T01:42:26.065 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640258 for osd.2 2024-04-19T01:42:26.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.110+0000 7f556893a640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-19T01:42:26.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.110+0000 7f556893a640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-19T01:42:26.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.110+0000 7f556893a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:26.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.110+0000 7f556893a640 1 -- 172.21.15.62:0/33811486 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f5564068710 con 0x7f55641380a0 2024-04-19T01:42:26.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.110+0000 7f5552ffd640 1 -- 172.21.15.62:0/33811486 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f556409c2e0 con 0x7f55641380a0 2024-04-19T01:42:26.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.110+0000 7f5552ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-19T01:42:26.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.110+0000 7f5552ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:26.111 INFO:teuthology.orchestra.run.smithi062.stdout:47244640258 2024-04-19T01:42:26.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.110+0000 7f556893a640 1 -- 172.21.15.62:0/33811486 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f55440364a0 msgr2=0x7f5544038960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:26.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.110+0000 7f556893a640 1 --2- 172.21.15.62:0/33811486 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f55440364a0 0x7f5544038960 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f555400b810 tx=0x7f5554008040 comp rx=0 tx=0).stop 2024-04-19T01:42:26.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.110+0000 7f556893a640 10 monclient: shutdown 2024-04-19T01:42:26.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.110+0000 7f556893a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:26.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.110+0000 7f556893a640 1 -- 172.21.15.62:0/33811486 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55641380a0 msgr2=0x7f55641363f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:26.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.110+0000 7f556893a640 1 --2- 172.21.15.62:0/33811486 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55641380a0 0x7f55641363f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f555c01cfb0 tx=0x7f555c0194b0 comp rx=0 tx=0).stop 2024-04-19T01:42:26.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.110+0000 7f556893a640 1 -- 172.21.15.62:0/33811486 shutdown_connections 2024-04-19T01:42:26.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.110+0000 7f556893a640 1 --2- 172.21.15.62:0/33811486 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f55440364a0 0x7f5544038960 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:26.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.110+0000 7f556893a640 1 --2- 172.21.15.62:0/33811486 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f55641380a0 0x7f55641363f0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:26.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.110+0000 7f556893a640 1 --2- 172.21.15.62:0/33811486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5564137780 0x7f556413e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:26.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.110+0000 7f556893a640 1 --2- 172.21.15.62:0/33811486 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55640a60b0 0x7f55641cd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:26.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.110+0000 7f556893a640 1 -- 172.21.15.62:0/33811486 >> 172.21.15.62:0/33811486 conn(0x7f55640aa1d0 msgr2=0x7f556409f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:26.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.110+0000 7f556893a640 1 -- 172.21.15.62:0/33811486 shutdown_connections 2024-04-19T01:42:26.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:26.110+0000 7f556893a640 1 -- 172.21.15.62:0/33811486 wait complete. 2024-04-19T01:42:26.127 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640258 for osd.1 2024-04-19T01:42:26.913 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-19T01:42:26.926 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-19T01:42:26.946 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-19T01:42:27.015 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-19T01:42:27.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d81ea640 10 monclient: get_monmap_and_config 2024-04-19T01:42:27.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d81ea640 10 monclient: build_initial_monmap 2024-04-19T01:42:27.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d81ea640 10 monclient: monmap: 2024-04-19T01:42:27.054 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:27.054 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.054 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:27.058583+0000 2024-04-19T01:42:27.054 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:27.058583+0000 2024-04-19T01:42:27.054 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:27.055 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.055 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:27.055 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:27.055 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:27.055 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:27.055 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:27.055 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d81ea640 1 Processor -- start 2024-04-19T01:42:27.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d81ea640 1 -- start start 2024-04-19T01:42:27.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d81ea640 10 monclient: init 2024-04-19T01:42:27.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d81ea640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:27.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d81ea640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-04-19T01:42:27.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d81ea640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa7d0137780 0x7fa7d0137b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d81ea640 10 monclient(hunting): picked mon.noname-c con 0x7fa7d0137780 addr v2:172.21.15.71:3300/0 2024-04-19T01:42:27.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d81ea640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa7d01380a0 0x7fa7d00a45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d81ea640 10 monclient(hunting): picked mon.noname-a con 0x7fa7d01380a0 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:27.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d81ea640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa7d00a60b0 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:27.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d81ea640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7d009c2e0 con 0x7fa7d00a60b0 2024-04-19T01:42:27.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d81ea640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d81ea640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fa7d009bd20 con 0x7fa7d01380a0 2024-04-19T01:42:27.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d81ea640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d81ea640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fa7d009c000 con 0x7fa7d0137780 2024-04-19T01:42:27.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d81ea640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:27.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d81ea640 10 monclient(hunting): authenticate will time out at 932.878071s 2024-04-19T01:42:27.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d6787640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fa7d00a60b0 0x7fa7d00a4b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:33564/0 (socket says 172.21.15.62:33564) 2024-04-19T01:42:27.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d6787640 1 -- 172.21.15.62:0/2993992170 learned_addr learned my addr 172.21.15.62:0/2993992170 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:42:27.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d5785640 1 --2- 172.21.15.62:0/2993992170 >> v2:172.21.15.62:3300/0 conn(0x7fa7d01380a0 0x7fa7d00a45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d5785640 10 monclient(hunting): get_auth_request con 0x7fa7d01380a0 auth_method 0 2024-04-19T01:42:27.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d5785640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:27.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d5785640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:27.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d5785640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:27.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d5785640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:27.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d5785640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:27.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d5785640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:27.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d5f86640 1 --2- 172.21.15.62:0/2993992170 >> v2:172.21.15.71:3300/0 conn(0x7fa7d0137780 0x7fa7d0137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d5f86640 10 monclient(hunting): get_auth_request con 0x7fa7d0137780 auth_method 0 2024-04-19T01:42:27.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d5f86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:27.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d5f86640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:27.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d5f86640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:27.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d5785640 10 monclient(hunting): handle_auth_done global_id 4266 payload 274 2024-04-19T01:42:27.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d5785640 1 -- 172.21.15.62:0/2993992170 >> v2:172.21.15.71:3300/0 conn(0x7fa7d0137780 msgr2=0x7fa7d0137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:27.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d5785640 1 --2- 172.21.15.62:0/2993992170 >> v2:172.21.15.71:3300/0 conn(0x7fa7d0137780 0x7fa7d0137b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d5785640 1 -- 172.21.15.62:0/2993992170 >> v1:172.21.15.62:6789/0 conn(0x7fa7d00a60b0 legacy=0x7fa7d00a4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:27.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d5785640 10 monclient: _finish_hunting 0 2024-04-19T01:42:27.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d5785640 1 monclient: found mon.noname-a 2024-04-19T01:42:27.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d5785640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:27.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d5785640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:27.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.054+0000 7fa7d5785640 1 -- 172.21.15.62:0/2993992170 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7d0073210 con 0x7fa7d01380a0 2024-04-19T01:42:27.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.058+0000 7fa7d5785640 1 -- 172.21.15.62:0/2993992170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7d01380a0 msgr2=0x7fa7d00a45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:27.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.058+0000 7fa7d5785640 1 --2- 172.21.15.62:0/2993992170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7d01380a0 0x7fa7d00a45d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa7d00a5a70 tx=0x7fa7c001c160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f553f660e15d6dd server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:27.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.058+0000 7fa7d4f84640 1 -- 172.21.15.62:0/2993992170 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c00b0070 con 0x7fa7d01380a0 2024-04-19T01:42:27.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.058+0000 7fa7d4f84640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:27.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.058+0000 7fa7d4f84640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:27.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.058+0000 7fa7d4f84640 10 monclient: dump: 2024-04-19T01:42:27.061 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:27.061 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.061 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.061 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.061 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:27.061 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.061 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:27.061 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:27.061 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:27.061 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.058+0000 7fa7d4f84640 10 monclient: _finish_auth 0 2024-04-19T01:42:27.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.058+0000 7fa7d4f84640 10 monclient: _check_auth_tickets 2024-04-19T01:42:27.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.058+0000 7fa7d4f84640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:27.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.058+0000 7fa7d4f84640 1 -- 172.21.15.62:0/2993992170 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7c00ab600 con 0x7fa7d01380a0 2024-04-19T01:42:27.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.058+0000 7fa7d4f84640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:27.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.058+0000 7fa7d4f84640 1 -- 172.21.15.62:0/2993992170 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c00ab9d0 con 0x7fa7d01380a0 2024-04-19T01:42:27.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.058+0000 7fa7d81ea640 5 monclient: authenticate success, global_id 4266 2024-04-19T01:42:27.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.058+0000 7fa7d81ea640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:27.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.058+0000 7fa7d81ea640 4 set_mon_vals no callback set 2024-04-19T01:42:27.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.058+0000 7fa7d4f84640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:27.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.058+0000 7fa7d4f84640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:27.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.058+0000 7fa7d4f84640 10 monclient: dump: 2024-04-19T01:42:27.062 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:27.062 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.062 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.062 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.063 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:27.063 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.063 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:27.063 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:27.063 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:27.063 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d81ea640 10 monclient: shutdown 2024-04-19T01:42:27.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d81ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:27.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d81ea640 1 -- 172.21.15.62:0/2993992170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7d01380a0 msgr2=0x7fa7d00a45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d81ea640 1 --2- 172.21.15.62:0/2993992170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7d01380a0 0x7fa7d00a45d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa7d00a5a70 tx=0x7fa7c001c160 comp rx=0 tx=0).stop 2024-04-19T01:42:27.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d81ea640 1 -- 172.21.15.62:0/2993992170 shutdown_connections 2024-04-19T01:42:27.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d81ea640 1 --2- 172.21.15.62:0/2993992170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7d01380a0 0x7fa7d00a45d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d81ea640 1 --2- 172.21.15.62:0/2993992170 >> v2:172.21.15.71:3300/0 conn(0x7fa7d0137780 0x7fa7d0137b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d81ea640 1 -- 172.21.15.62:0/2993992170 >> 172.21.15.62:0/2993992170 conn(0x7fa7d00aa1d0 msgr2=0x7fa7d00a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:27.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d81ea640 1 -- 172.21.15.62:0/2993992170 shutdown_connections 2024-04-19T01:42:27.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d81ea640 1 -- 172.21.15.62:0/2993992170 wait complete. 2024-04-19T01:42:27.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d81ea640 10 monclient: build_initial_monmap 2024-04-19T01:42:27.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d81ea640 10 monclient: monmap: 2024-04-19T01:42:27.067 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:27.067 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:27.067 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:27.067 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:27.067 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:27.067 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.067 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:27.067 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:27.067 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:27.067 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d81ea640 1 Processor -- start 2024-04-19T01:42:27.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d81ea640 1 -- start start 2024-04-19T01:42:27.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d81ea640 10 monclient: init 2024-04-19T01:42:27.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d81ea640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:27.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d81ea640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-19T01:42:27.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d81ea640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7d00a60b0 0x7fa7d01364d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d81ea640 10 monclient(hunting): picked mon.noname-c con 0x7fa7d00a60b0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:27.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d81ea640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa7d0137780 0x7fa7d013c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d81ea640 10 monclient(hunting): picked mon.noname-b con 0x7fa7d0137780 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:27.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d81ea640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7d01380a0 0x7fa7d013c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d81ea640 10 monclient(hunting): picked mon.noname-a con 0x7fa7d01380a0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:27.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d81ea640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d81ea640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d0142170 con 0x7fa7d01380a0 2024-04-19T01:42:27.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d81ea640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d81ea640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d00a5430 con 0x7fa7d0137780 2024-04-19T01:42:27.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d81ea640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d81ea640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fa7d01422f0 con 0x7fa7d00a60b0 2024-04-19T01:42:27.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d81ea640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:27.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d81ea640 10 monclient(hunting): authenticate will time out at 932.886040s 2024-04-19T01:42:27.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d5f86640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7d00a60b0 0x7fa7d01364d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d5f86640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7d00a60b0 0x7fa7d01364d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60392/0 (socket says 172.21.15.62:60392) 2024-04-19T01:42:27.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d5f86640 1 -- 172.21.15.62:0/3835028680 learned_addr learned my addr 172.21.15.62:0/3835028680 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:27.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d5f86640 10 monclient(hunting): get_auth_request con 0x7fa7d00a60b0 auth_method 0 2024-04-19T01:42:27.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d5f86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:27.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d5f86640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:27.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d5f86640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:27.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d6787640 1 --2- 172.21.15.62:0/3835028680 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7d01380a0 0x7fa7d013c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d6787640 10 monclient(hunting): get_auth_request con 0x7fa7d01380a0 auth_method 0 2024-04-19T01:42:27.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d6787640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:27.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d6787640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:27.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d6787640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:27.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d5785640 1 --2- 172.21.15.62:0/3835028680 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa7d0137780 0x7fa7d013c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d5785640 10 monclient(hunting): get_auth_request con 0x7fa7d0137780 auth_method 0 2024-04-19T01:42:27.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d5785640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:27.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d5785640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:27.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d5785640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:27.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d5f86640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:27.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d5f86640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:27.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d5f86640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:27.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d6787640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:27.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d6787640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:27.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d6787640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:27.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d6787640 10 monclient(hunting): handle_auth_done global_id 4269 payload 824 2024-04-19T01:42:27.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d6787640 1 -- 172.21.15.62:0/3835028680 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7d00a60b0 msgr2=0x7fa7d01364d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d6787640 1 --2- 172.21.15.62:0/3835028680 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7d00a60b0 0x7fa7d01364d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d6787640 1 -- 172.21.15.62:0/3835028680 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa7d0137780 msgr2=0x7fa7d013c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d6787640 1 --2- 172.21.15.62:0/3835028680 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa7d0137780 0x7fa7d013c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d6787640 10 monclient: _finish_hunting 0 2024-04-19T01:42:27.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d6787640 1 monclient: found mon.noname-a 2024-04-19T01:42:27.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d6787640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:27.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d6787640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:27.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d6787640 1 -- 172.21.15.62:0/3835028680 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7d00a3420 con 0x7fa7d01380a0 2024-04-19T01:42:27.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.062+0000 7fa7d5f86640 1 --2- 172.21.15.62:0/3835028680 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7d00a60b0 0x7fa7d01364d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:42:27.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.066+0000 7fa7d6787640 1 -- 172.21.15.62:0/3835028680 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7d01380a0 msgr2=0x7fa7d013c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:27.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.066+0000 7fa7d6787640 1 --2- 172.21.15.62:0/3835028680 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7d01380a0 0x7fa7d013c810 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa7c800caf0 tx=0x7fa7c800cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:27.071 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-19T01:42:27.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.066+0000 7fa7c67fc640 1 -- 172.21.15.62:0/3835028680 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c800ade0 con 0x7fa7d01380a0 2024-04-19T01:42:27.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.066+0000 7fa7c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:27.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.066+0000 7fa7c67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:27.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.066+0000 7fa7c67fc640 10 monclient: dump: 2024-04-19T01:42:27.074 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:27.074 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.074 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.074 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.074 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:27.074 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.074 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:27.074 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:27.074 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:27.074 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.066+0000 7fa7c67fc640 10 monclient: _finish_auth 0 2024-04-19T01:42:27.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.066+0000 7fa7c67fc640 10 monclient: _check_auth_tickets 2024-04-19T01:42:27.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.066+0000 7fa7c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:27.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.066+0000 7fa7c67fc640 1 -- 172.21.15.62:0/3835028680 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7c8012400 con 0x7fa7d01380a0 2024-04-19T01:42:27.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.066+0000 7fa7d81ea640 5 monclient: authenticate success, global_id 4269 2024-04-19T01:42:27.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.066+0000 7fa7c67fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:27.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.066+0000 7fa7c67fc640 1 -- 172.21.15.62:0/3835028680 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c80127f0 con 0x7fa7d01380a0 2024-04-19T01:42:27.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.066+0000 7fa7c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:27.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.066+0000 7fa7c67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:27.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.066+0000 7fa7c67fc640 10 monclient: dump: 2024-04-19T01:42:27.075 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:27.075 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.075 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.075 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.075 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:27.075 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.075 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:27.076 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:27.076 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:27.076 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.066+0000 7fa7c7fff640 4 set_mon_vals no callback set 2024-04-19T01:42:27.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.066+0000 7fa7d81ea640 10 monclient: _renew_subs 2024-04-19T01:42:27.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.066+0000 7fa7d81ea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:27.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.066+0000 7fa7d81ea640 1 -- 172.21.15.62:0/3835028680 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7d00a2300 con 0x7fa7d01380a0 2024-04-19T01:42:27.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.066+0000 7fa7d81ea640 10 monclient: _renew_subs 2024-04-19T01:42:27.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.066+0000 7fa7d81ea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:27.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.066+0000 7fa7d81ea640 1 -- 172.21.15.62:0/3835028680 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7d0120c80 con 0x7fa7d01380a0 2024-04-19T01:42:27.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.066+0000 7fa7c67fc640 1 -- 172.21.15.62:0/3835028680 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fa7c80129d0 con 0x7fa7d01380a0 2024-04-19T01:42:27.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.066+0000 7fa7c67fc640 1 --2- 172.21.15.62:0/3835028680 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa7b4036450 0x7fa7b4038910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.066+0000 7fa7c67fc640 1 -- 172.21.15.62:0/3835028680 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4346+0+0 (secure 0 0 0) 0x7fa7c80498d0 con 0x7fa7d01380a0 2024-04-19T01:42:27.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.066+0000 7fa7d81ea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:27.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.066+0000 7fa7d81ea640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:27.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.066+0000 7fa7d81ea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:27.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.066+0000 7fa7d81ea640 1 -- 172.21.15.62:0/3835028680 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7d0068ab0 con 0x7fa7d01380a0 2024-04-19T01:42:27.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.070+0000 7fa7d5f86640 1 --2- 172.21.15.62:0/3835028680 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa7b4036450 0x7fa7b4038910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.070+0000 7fa7d5f86640 10 monclient: get_auth_request con 0x7fa7b4036450 auth_method 0 2024-04-19T01:42:27.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.070+0000 7fa7d5f86640 1 --2- 172.21.15.62:0/3835028680 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa7b4036450 0x7fa7b4038910 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa7d013db70 tx=0x7fa7cc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:27.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.078+0000 7fa7c67fc640 1 -- 172.21.15.62:0/3835028680 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa7c801a360 con 0x7fa7d01380a0 2024-04-19T01:42:27.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.078+0000 7fa7c67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:27.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.078+0000 7fa7c67fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:27.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.078+0000 7facd6f6b640 10 monclient: get_monmap_and_config 2024-04-19T01:42:27.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.078+0000 7facd6f6b640 10 monclient: build_initial_monmap 2024-04-19T01:42:27.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.078+0000 7facd6f6b640 10 monclient: monmap: 2024-04-19T01:42:27.080 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:27.080 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.080 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:27.084695+0000 2024-04-19T01:42:27.080 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:27.084695+0000 2024-04-19T01:42:27.080 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:27.080 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.080 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:27.080 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:27.081 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:27.081 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:27.081 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:27.081 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.078+0000 7facd6f6b640 1 Processor -- start 2024-04-19T01:42:27.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.078+0000 7facd6f6b640 1 -- start start 2024-04-19T01:42:27.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.078+0000 7facd6f6b640 10 monclient: init 2024-04-19T01:42:27.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.078+0000 7facd6f6b640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:27.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.078+0000 7facd6f6b640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-04-19T01:42:27.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.078+0000 7facd6f6b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7facd01355e0 addr v1:172.21.15.71:6789/0 2024-04-19T01:42:27.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.078+0000 7facd6f6b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7facd00726e0 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:27.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.078+0000 7facd6f6b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7facd0072b00 0x7facd013dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.078+0000 7facd6f6b640 10 monclient(hunting): picked mon.noname-b con 0x7facd0072b00 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:27.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.078+0000 7facd6f6b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facd00716c0 con 0x7facd00726e0 2024-04-19T01:42:27.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.078+0000 7facd6f6b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facd00719a0 con 0x7facd01355e0 2024-04-19T01:42:27.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.078+0000 7facd6f6b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.078+0000 7facd6f6b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7facd01360d0 con 0x7facd0072b00 2024-04-19T01:42:27.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.078+0000 7facd6f6b640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:27.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.078+0000 7facd6f6b640 10 monclient(hunting): authenticate will time out at 932.903889s 2024-04-19T01:42:27.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.082+0000 7facd5508640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7facd0072b00 0x7facd013dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.082+0000 7facd5508640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7facd0072b00 0x7facd013dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60396/0 (socket says 172.21.15.62:60396) 2024-04-19T01:42:27.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.082+0000 7facd5508640 1 -- 172.21.15.62:0/540032431 learned_addr learned my addr 172.21.15.62:0/540032431 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:27.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.082+0000 7facd5508640 10 monclient(hunting): get_auth_request con 0x7facd0072b00 auth_method 0 2024-04-19T01:42:27.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.082+0000 7facd5508640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:27.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.082+0000 7facd5508640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:27.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.082+0000 7facd5508640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:27.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.082+0000 7facd5508640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:27.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.082+0000 7facd5508640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:27.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.082+0000 7facd5508640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:27.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.082+0000 7facd5508640 10 monclient(hunting): handle_auth_done global_id 4271 payload 274 2024-04-19T01:42:27.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.082+0000 7facd5508640 1 -- 172.21.15.62:0/540032431 >> v1:172.21.15.71:6789/0 conn(0x7facd01355e0 legacy=0x7facd01359c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:27.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.082+0000 7facd5508640 1 -- 172.21.15.62:0/540032431 >> v1:172.21.15.62:6789/0 conn(0x7facd00726e0 legacy=0x7facd013a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:27.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.082+0000 7facd5508640 10 monclient: _finish_hunting 0 2024-04-19T01:42:27.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.082+0000 7facd5508640 1 monclient: found mon.noname-b 2024-04-19T01:42:27.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.082+0000 7facd5508640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:27.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.082+0000 7facd5508640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:42:27.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.082+0000 7facd5508640 1 -- 172.21.15.62:0/540032431 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facd0072f50 con 0x7facd0072b00 2024-04-19T01:42:27.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.086+0000 7facd5508640 1 -- 172.21.15.62:0/540032431 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7facd0072b00 msgr2=0x7facd013dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:27.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.086+0000 7facd5508640 1 --2- 172.21.15.62:0/540032431 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7facd0072b00 0x7facd013dbb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7facc8009e80 tx=0x7facc801c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6307561be130a445 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:27.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.086+0000 7faccf7fe640 1 -- 172.21.15.62:0/540032431 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc80a6500 con 0x7facd0072b00 2024-04-19T01:42:27.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.086+0000 7faccf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:27.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.086+0000 7faccf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:42:27.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.086+0000 7faccf7fe640 10 monclient: dump: 2024-04-19T01:42:27.088 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:27.088 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.088 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.088 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.088 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:27.088 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.088 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:27.088 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:27.088 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:27.088 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.086+0000 7faccf7fe640 10 monclient: _finish_auth 0 2024-04-19T01:42:27.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.086+0000 7faccf7fe640 10 monclient: _check_auth_tickets 2024-04-19T01:42:27.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.086+0000 7faccf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:27.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.086+0000 7facd6f6b640 5 monclient: authenticate success, global_id 4271 2024-04-19T01:42:27.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.086+0000 7facd6f6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:42:27.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.086+0000 7faccf7fe640 1 -- 172.21.15.62:0/540032431 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facc80a66a0 con 0x7facd0072b00 2024-04-19T01:42:27.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.086+0000 7faccf7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:27.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.086+0000 7facd6f6b640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:27.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.086+0000 7facd6f6b640 4 set_mon_vals no callback set 2024-04-19T01:42:27.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.086+0000 7facd6f6b640 10 monclient: shutdown 2024-04-19T01:42:27.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.086+0000 7facd6f6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:27.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.086+0000 7facd6f6b640 1 -- 172.21.15.62:0/540032431 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7facd0072b00 msgr2=0x7facd013dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.086+0000 7facd6f6b640 1 --2- 172.21.15.62:0/540032431 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7facd0072b00 0x7facd013dbb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7facc8009e80 tx=0x7facc801c910 comp rx=0 tx=0).stop 2024-04-19T01:42:27.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.086+0000 7facd6f6b640 1 -- 172.21.15.62:0/540032431 shutdown_connections 2024-04-19T01:42:27.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.086+0000 7facd6f6b640 1 --2- 172.21.15.62:0/540032431 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7facd0072b00 0x7facd013dbb0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.086+0000 7facd6f6b640 1 -- 172.21.15.62:0/540032431 >> 172.21.15.62:0/540032431 conn(0x7facd00a8030 msgr2=0x7facd00a8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:27.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.086+0000 7facd6f6b640 1 -- 172.21.15.62:0/540032431 shutdown_connections 2024-04-19T01:42:27.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.086+0000 7facd6f6b640 1 -- 172.21.15.62:0/540032431 wait complete. 2024-04-19T01:42:27.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.086+0000 7facd6f6b640 10 monclient: build_initial_monmap 2024-04-19T01:42:27.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.086+0000 7facd6f6b640 10 monclient: monmap: 2024-04-19T01:42:27.092 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:27.092 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:27.092 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:27.092 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:27.092 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:27.092 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.092 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:27.092 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:27.092 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:27.092 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd6f6b640 1 Processor -- start 2024-04-19T01:42:27.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd6f6b640 1 -- start start 2024-04-19T01:42:27.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd6f6b640 10 monclient: init 2024-04-19T01:42:27.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd6f6b640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:27.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd6f6b640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-19T01:42:27.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd6f6b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7facd00726e0 0x7facd01d35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd6f6b640 10 monclient(hunting): picked mon.noname-a con 0x7facd00726e0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:27.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd6f6b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7facd0072b00 0x7facd0143c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd6f6b640 10 monclient(hunting): picked mon.noname-c con 0x7facd0072b00 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:27.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd6f6b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7facd01355e0 0x7facd01268c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd6f6b640 10 monclient(hunting): picked mon.noname-b con 0x7facd01355e0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:27.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd6f6b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd6f6b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7facd006c560 con 0x7facd00726e0 2024-04-19T01:42:27.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd6f6b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd6f6b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7facd006c3e0 con 0x7facd01355e0 2024-04-19T01:42:27.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd6f6b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd6f6b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7facd006c6e0 con 0x7facd0072b00 2024-04-19T01:42:27.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd6f6b640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:27.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd6f6b640 10 monclient(hunting): authenticate will time out at 932.912597s 2024-04-19T01:42:27.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7faccffff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7facd0072b00 0x7facd0143c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7faccffff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7facd0072b00 0x7facd0143c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60406/0 (socket says 172.21.15.62:60406) 2024-04-19T01:42:27.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7faccffff640 1 -- 172.21.15.62:0/1047736057 learned_addr learned my addr 172.21.15.62:0/1047736057 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:27.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7faccffff640 10 monclient(hunting): get_auth_request con 0x7facd0072b00 auth_method 0 2024-04-19T01:42:27.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7faccffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:27.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7faccffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:27.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7faccffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:27.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd4d07640 1 --2- 172.21.15.62:0/1047736057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7facd00726e0 0x7facd01d35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd4d07640 10 monclient(hunting): get_auth_request con 0x7facd00726e0 auth_method 0 2024-04-19T01:42:27.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd4d07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:27.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd4d07640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:27.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd4d07640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:27.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7faccffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:27.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7faccffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:27.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7faccffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:27.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd4d07640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:27.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd4d07640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:27.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd4d07640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:27.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd5508640 1 --2- 172.21.15.62:0/1047736057 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7facd01355e0 0x7facd01268c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd5508640 10 monclient(hunting): get_auth_request con 0x7facd01355e0 auth_method 0 2024-04-19T01:42:27.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd5508640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:27.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd5508640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:27.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd5508640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:27.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd4d07640 10 monclient(hunting): handle_auth_done global_id 4272 payload 824 2024-04-19T01:42:27.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd4d07640 1 -- 172.21.15.62:0/1047736057 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7facd0072b00 msgr2=0x7facd0143c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd4d07640 1 --2- 172.21.15.62:0/1047736057 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7facd0072b00 0x7facd0143c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd4d07640 1 -- 172.21.15.62:0/1047736057 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7facd01355e0 msgr2=0x7facd01268c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd4d07640 1 --2- 172.21.15.62:0/1047736057 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7facd01355e0 0x7facd01268c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd4d07640 10 monclient: _finish_hunting 0 2024-04-19T01:42:27.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd4d07640 1 monclient: found mon.noname-a 2024-04-19T01:42:27.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd4d07640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:27.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd4d07640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:27.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd4d07640 1 -- 172.21.15.62:0/1047736057 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facd0140400 con 0x7facd00726e0 2024-04-19T01:42:27.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7faccffff640 1 --2- 172.21.15.62:0/1047736057 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7facd0072b00 0x7facd0143c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:42:27.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd5508640 1 --2- 172.21.15.62:0/1047736057 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7facd01355e0 0x7facd01268c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T01:42:27.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f445b11d640 10 monclient: get_monmap_and_config 2024-04-19T01:42:27.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f445b11d640 10 monclient: build_initial_monmap 2024-04-19T01:42:27.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f445b11d640 10 monclient: monmap: 2024-04-19T01:42:27.098 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:27.098 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.099 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:27.095806+0000 2024-04-19T01:42:27.099 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:27.095806+0000 2024-04-19T01:42:27.099 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:27.099 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.099 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:27.099 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:27.099 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:27.099 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:27.099 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:27.099 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f445b11d640 1 Processor -- start 2024-04-19T01:42:27.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f445b11d640 1 -- start start 2024-04-19T01:42:27.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f445b11d640 10 monclient: init 2024-04-19T01:42:27.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f445b11d640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:27.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f445b11d640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-04-19T01:42:27.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f445b11d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4454137780 0x7f4454137b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f445b11d640 10 monclient(hunting): picked mon.noname-a con 0x7f4454137780 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:27.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f445b11d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f44541380a0 addr v1:172.21.15.71:6789/0 2024-04-19T01:42:27.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f445b11d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f44540a60b0 0x7f44540a4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f445b11d640 10 monclient(hunting): picked mon.noname-b con 0x7f44540a60b0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:27.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f445b11d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f445409c2e0 con 0x7f44541380a0 2024-04-19T01:42:27.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f445b11d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f445b11d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f445409bd20 con 0x7f4454137780 2024-04-19T01:42:27.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f445b11d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f445b11d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f445409c000 con 0x7f44540a60b0 2024-04-19T01:42:27.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f445b11d640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:27.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f445b11d640 10 monclient(hunting): authenticate will time out at 932.915067s 2024-04-19T01:42:27.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f44596ba640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f44540a60b0 0x7f44540a4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f44596ba640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f44540a60b0 0x7f44540a4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60412/0 (socket says 172.21.15.62:60412) 2024-04-19T01:42:27.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f44596ba640 1 -- 172.21.15.62:0/3176697584 learned_addr learned my addr 172.21.15.62:0/3176697584 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:27.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f44596ba640 10 monclient(hunting): get_auth_request con 0x7f44540a60b0 auth_method 0 2024-04-19T01:42:27.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f44596ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:27.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f44596ba640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:27.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f44596ba640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:27.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f44596ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:27.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f44596ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:27.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f44596ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:27.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f44596ba640 10 monclient(hunting): handle_auth_done global_id 4277 payload 274 2024-04-19T01:42:27.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f44596ba640 1 -- 172.21.15.62:0/3176697584 >> v2:172.21.15.62:3300/0 conn(0x7f4454137780 msgr2=0x7f4454137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:27.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f44596ba640 1 --2- 172.21.15.62:0/3176697584 >> v2:172.21.15.62:3300/0 conn(0x7f4454137780 0x7f4454137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd4d07640 1 -- 172.21.15.62:0/1047736057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7facd00726e0 msgr2=0x7facd01d35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:27.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd4d07640 1 --2- 172.21.15.62:0/1047736057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7facd00726e0 0x7facd01d35c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7facc400c3e0 tx=0x7facc400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:27.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f44596ba640 1 -- 172.21.15.62:0/3176697584 >> v1:172.21.15.71:6789/0 conn(0x7f44541380a0 legacy=0x7f44540a45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:27.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f44596ba640 10 monclient: _finish_hunting 0 2024-04-19T01:42:27.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f44596ba640 1 monclient: found mon.noname-b 2024-04-19T01:42:27.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f44596ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:27.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f44596ba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:42:27.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7f44596ba640 1 -- 172.21.15.62:0/3176697584 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4454073210 con 0x7f44540a60b0 2024-04-19T01:42:27.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7faccd7fa640 1 -- 172.21.15.62:0/1047736057 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc40119a0 con 0x7facd00726e0 2024-04-19T01:42:27.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7faccd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:27.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7faccd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:27.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7faccd7fa640 10 monclient: dump: 2024-04-19T01:42:27.104 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:27.104 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.105 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.105 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.105 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:27.105 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.105 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:27.105 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:27.105 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:27.105 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7faccd7fa640 10 monclient: _finish_auth 0 2024-04-19T01:42:27.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7faccd7fa640 10 monclient: _check_auth_tickets 2024-04-19T01:42:27.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7faccd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:27.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7faccd7fa640 1 -- 172.21.15.62:0/1047736057 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facc4017070 con 0x7facd00726e0 2024-04-19T01:42:27.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facd6f6b640 5 monclient: authenticate success, global_id 4272 2024-04-19T01:42:27.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7faccd7fa640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:27.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.090+0000 7facceffd640 4 set_mon_vals no callback set 2024-04-19T01:42:27.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.094+0000 7facd6f6b640 10 monclient: _renew_subs 2024-04-19T01:42:27.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.094+0000 7faccd7fa640 1 -- 172.21.15.62:0/1047736057 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc4011dc0 con 0x7facd00726e0 2024-04-19T01:42:27.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.094+0000 7facd6f6b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:27.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.094+0000 7facd6f6b640 1 -- 172.21.15.62:0/1047736057 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facd0128430 con 0x7facd00726e0 2024-04-19T01:42:27.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.094+0000 7facd6f6b640 10 monclient: _renew_subs 2024-04-19T01:42:27.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.094+0000 7facd6f6b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:27.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.094+0000 7facd6f6b640 1 -- 172.21.15.62:0/1047736057 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facd013d350 con 0x7facd00726e0 2024-04-19T01:42:27.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.094+0000 7facd6f6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:27.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.094+0000 7facd6f6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:27.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.094+0000 7facd6f6b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:27.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.094+0000 7facd6f6b640 1 -- 172.21.15.62:0/1047736057 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facd0068ab0 con 0x7facd00726e0 2024-04-19T01:42:27.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.094+0000 7faccd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:27.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.094+0000 7faccd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:27.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.094+0000 7faccd7fa640 10 monclient: dump: 2024-04-19T01:42:27.106 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:27.107 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.107 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.107 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.107 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:27.107 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.107 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:27.107 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:27.107 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:27.107 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.094+0000 7faccd7fa640 1 -- 172.21.15.62:0/1047736057 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7facc401c460 con 0x7facd00726e0 2024-04-19T01:42:27.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.094+0000 7faccd7fa640 1 --2- 172.21.15.62:0/1047736057 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7facb4036420 0x7facb40388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.094+0000 7faccd7fa640 1 -- 172.21.15.62:0/1047736057 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4346+0+0 (secure 0 0 0) 0x7facc4049120 con 0x7facd00726e0 2024-04-19T01:42:27.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.094+0000 7faccffff640 1 --2- 172.21.15.62:0/1047736057 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7facb4036420 0x7facb40388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.094+0000 7faccffff640 10 monclient: get_auth_request con 0x7facb4036420 auth_method 0 2024-04-19T01:42:27.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.094+0000 7f44596ba640 1 -- 172.21.15.62:0/3176697584 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f44540a60b0 msgr2=0x7f44540a4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:27.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.094+0000 7f44596ba640 1 --2- 172.21.15.62:0/3176697584 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f44540a60b0 0x7f44540a4ce0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f444c009e50 tx=0x7f444c01c710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fb1b8303aa08218 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:27.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f44537fe640 1 -- 172.21.15.62:0/3176697584 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f444c0ac020 con 0x7f44540a60b0 2024-04-19T01:42:27.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f44537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:27.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f44537fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:42:27.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f44537fe640 10 monclient: dump: 2024-04-19T01:42:27.109 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:27.109 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.109 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.109 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.109 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:27.109 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.109 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:27.109 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:27.109 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:27.110 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f44537fe640 10 monclient: _finish_auth 0 2024-04-19T01:42:27.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f44537fe640 10 monclient: _check_auth_tickets 2024-04-19T01:42:27.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f44537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:27.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f44537fe640 1 -- 172.21.15.62:0/3176697584 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f444c0a6470 con 0x7f44540a60b0 2024-04-19T01:42:27.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f44537fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:27.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f44537fe640 1 -- 172.21.15.62:0/3176697584 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f444c0a6820 con 0x7f44540a60b0 2024-04-19T01:42:27.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f44537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:27.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f44537fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:27.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f44537fe640 10 monclient: dump: 2024-04-19T01:42:27.110 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:27.110 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.110 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.110 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.110 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:27.111 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.111 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:27.111 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:27.111 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:27.111 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 5 monclient: authenticate success, global_id 4277 2024-04-19T01:42:27.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:27.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 4 set_mon_vals no callback set 2024-04-19T01:42:27.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 10 monclient: shutdown 2024-04-19T01:42:27.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:27.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 1 -- 172.21.15.62:0/3176697584 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f44540a60b0 msgr2=0x7f44540a4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 1 --2- 172.21.15.62:0/3176697584 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f44540a60b0 0x7f44540a4ce0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f444c009e50 tx=0x7f444c01c710 comp rx=0 tx=0).stop 2024-04-19T01:42:27.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 1 -- 172.21.15.62:0/3176697584 shutdown_connections 2024-04-19T01:42:27.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 1 --2- 172.21.15.62:0/3176697584 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f44540a60b0 0x7f44540a4ce0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 1 --2- 172.21.15.62:0/3176697584 >> v2:172.21.15.62:3300/0 conn(0x7f4454137780 0x7f4454137b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 1 -- 172.21.15.62:0/3176697584 >> 172.21.15.62:0/3176697584 conn(0x7f44540aa1d0 msgr2=0x7f44540a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:27.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 1 -- 172.21.15.62:0/3176697584 shutdown_connections 2024-04-19T01:42:27.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 1 -- 172.21.15.62:0/3176697584 wait complete. 2024-04-19T01:42:27.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7faccffff640 1 --2- 172.21.15.62:0/1047736057 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7facb4036420 0x7facb40388e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7facd0128a30 tx=0x7facbc003700 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:27.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 10 monclient: build_initial_monmap 2024-04-19T01:42:27.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 10 monclient: monmap: 2024-04-19T01:42:27.114 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:27.114 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:27.114 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:27.114 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:27.114 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:27.115 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.115 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:27.115 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:27.115 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:27.115 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 1 Processor -- start 2024-04-19T01:42:27.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 1 -- start start 2024-04-19T01:42:27.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 10 monclient: init 2024-04-19T01:42:27.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:27.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-19T01:42:27.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f44540a60b0 0x7f445413ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 10 monclient(hunting): picked mon.noname-c con 0x7f44540a60b0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:27.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4454137780 0x7f44541384b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 10 monclient(hunting): picked mon.noname-a con 0x7f4454137780 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:27.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44541380a0 0x7f44541389f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 10 monclient(hunting): picked mon.noname-b con 0x7f44541380a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:27.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4454142170 con 0x7f4454137780 2024-04-19T01:42:27.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f44540a5430 con 0x7f44541380a0 2024-04-19T01:42:27.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f44541422f0 con 0x7f44540a60b0 2024-04-19T01:42:27.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:27.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f445b11d640 10 monclient(hunting): authenticate will time out at 932.922920s 2024-04-19T01:42:27.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f4458eb9640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f44540a60b0 0x7f445413ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f4453fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4454137780 0x7f44541384b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f4458eb9640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f44540a60b0 0x7f445413ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60424/0 (socket says 172.21.15.62:60424) 2024-04-19T01:42:27.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f4453fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4454137780 0x7f44541384b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36350/0 (socket says 172.21.15.62:36350) 2024-04-19T01:42:27.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f4458eb9640 1 -- 172.21.15.62:0/1134699292 learned_addr learned my addr 172.21.15.62:0/1134699292 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:27.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f4458eb9640 10 monclient(hunting): get_auth_request con 0x7f44540a60b0 auth_method 0 2024-04-19T01:42:27.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f4458eb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:27.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f4458eb9640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:27.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f4458eb9640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:27.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f4453fff640 10 monclient(hunting): get_auth_request con 0x7f4454137780 auth_method 0 2024-04-19T01:42:27.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f4453fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:27.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f4453fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:27.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f4453fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:27.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f4458eb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:27.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f4458eb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:27.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f4458eb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:27.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f4453fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:27.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f4453fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:27.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f4453fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:27.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f4458eb9640 10 monclient(hunting): handle_auth_done global_id 4280 payload 824 2024-04-19T01:42:27.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f4458eb9640 1 -- 172.21.15.62:0/1134699292 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44541380a0 msgr2=0x7f44541389f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f44596ba640 1 --2- 172.21.15.62:0/1134699292 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44541380a0 0x7f44541389f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f4458eb9640 1 --2- 172.21.15.62:0/1134699292 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44541380a0 0x7f44541389f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f4458eb9640 1 -- 172.21.15.62:0/1134699292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4454137780 msgr2=0x7f44541384b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f4458eb9640 1 --2- 172.21.15.62:0/1134699292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4454137780 0x7f44541384b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f4458eb9640 10 monclient: _finish_hunting 0 2024-04-19T01:42:27.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f4458eb9640 1 monclient: found mon.noname-c 2024-04-19T01:42:27.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f4458eb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:27.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f4458eb9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:42:27.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f4458eb9640 1 -- 172.21.15.62:0/1134699292 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f445413b7f0 con 0x7f44540a60b0 2024-04-19T01:42:27.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f44596ba640 10 monclient: get_auth_request con 0x7f44541380a0 auth_method 0 2024-04-19T01:42:27.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.098+0000 7f44596ba640 1 --2- 172.21.15.62:0/1134699292 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44541380a0 0x7f44541389f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:42:27.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7faccd7fa640 1 -- 172.21.15.62:0/1047736057 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7facc401c9e0 con 0x7facd00726e0 2024-04-19T01:42:27.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7faccd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:27.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7faccd7fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:27.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7f4458eb9640 1 -- 172.21.15.62:0/1134699292 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f44540a60b0 msgr2=0x7f445413ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:27.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7f4458eb9640 1 --2- 172.21.15.62:0/1134699292 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f44540a60b0 0x7f445413ef00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f444800b550 tx=0x7f444800ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:27.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7f44517fa640 1 -- 172.21.15.62:0/1134699292 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4448014070 con 0x7f44540a60b0 2024-04-19T01:42:27.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7f44517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:27.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7f44517fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:27.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7f44517fa640 10 monclient: dump: 2024-04-19T01:42:27.121 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:27.121 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.121 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.121 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.121 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:27.121 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.121 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:27.121 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:27.122 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:27.122 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7f44517fa640 10 monclient: _finish_auth 0 2024-04-19T01:42:27.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7f44517fa640 10 monclient: _check_auth_tickets 2024-04-19T01:42:27.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7f44517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:27.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7f44517fa640 1 -- 172.21.15.62:0/1134699292 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4448010aa0 con 0x7f44540a60b0 2024-04-19T01:42:27.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7f44517fa640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:27.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7f44517fa640 1 -- 172.21.15.62:0/1134699292 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f444800f3c0 con 0x7f44540a60b0 2024-04-19T01:42:27.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7f44517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:27.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7f44517fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:27.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7f44517fa640 10 monclient: dump: 2024-04-19T01:42:27.122 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:27.122 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.122 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.123 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.123 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:27.123 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.123 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:27.123 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:27.123 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:27.123 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7f445b11d640 5 monclient: authenticate success, global_id 4280 2024-04-19T01:42:27.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7f445b11d640 10 monclient: _renew_subs 2024-04-19T01:42:27.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7f4452ffd640 4 set_mon_vals no callback set 2024-04-19T01:42:27.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7f445b11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:27.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7f445b11d640 1 -- 172.21.15.62:0/1134699292 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f445413bad0 con 0x7f44540a60b0 2024-04-19T01:42:27.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7f445b11d640 10 monclient: _renew_subs 2024-04-19T01:42:27.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7f445b11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:27.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7f445b11d640 1 -- 172.21.15.62:0/1134699292 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4454139b30 con 0x7f44540a60b0 2024-04-19T01:42:27.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7f445b11d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:27.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7f445b11d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:27.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7f445b11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:27.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7f445b11d640 1 -- 172.21.15.62:0/1134699292 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4454068ab0 con 0x7f44540a60b0 2024-04-19T01:42:27.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7f44517fa640 1 -- 172.21.15.62:0/1134699292 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f4448017070 con 0x7f44540a60b0 2024-04-19T01:42:27.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7f44517fa640 1 --2- 172.21.15.62:0/1134699292 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4438036470 0x7f4438038930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.102+0000 7f44517fa640 1 -- 172.21.15.62:0/1134699292 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4346+0+0 (secure 0 0 0) 0x7f4448049140 con 0x7f44540a60b0 2024-04-19T01:42:27.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.110+0000 7f4453fff640 1 --2- 172.21.15.62:0/1134699292 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4438036470 0x7f4438038930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.110+0000 7f4453fff640 10 monclient: get_auth_request con 0x7f4438036470 auth_method 0 2024-04-19T01:42:27.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.110+0000 7f4453fff640 1 --2- 172.21.15.62:0/1134699292 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4438036470 0x7f4438038930 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f44540a1380 tx=0x7f4440007e40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:27.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.114+0000 7f44517fa640 1 -- 172.21.15.62:0/1134699292 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4448018570 con 0x7f44540a60b0 2024-04-19T01:42:27.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.114+0000 7f44517fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:27.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.114+0000 7f44517fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:27.128 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-19T01:42:27.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ef95a640 10 monclient: get_monmap_and_config 2024-04-19T01:42:27.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ef95a640 10 monclient: build_initial_monmap 2024-04-19T01:42:27.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ef95a640 10 monclient: monmap: 2024-04-19T01:42:27.178 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:27.178 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.178 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:27.182190+0000 2024-04-19T01:42:27.178 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:27.182190+0000 2024-04-19T01:42:27.178 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:27.178 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.178 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:27.178 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:27.178 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:27.178 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:27.178 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:27.178 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ef95a640 1 Processor -- start 2024-04-19T01:42:27.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ef95a640 1 -- start start 2024-04-19T01:42:27.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ef95a640 10 monclient: init 2024-04-19T01:42:27.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ef95a640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:27.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ef95a640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-04-19T01:42:27.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ef95a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb6e8137780 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:27.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ef95a640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb6e80a60b0 0x7fb6e813c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ef95a640 10 monclient(hunting): picked mon.noname-a con 0x7fb6e80a60b0 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:27.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ef95a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb6e813cc60 addr v1:172.21.15.71:6789/0 2024-04-19T01:42:27.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ef95a640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6e809c2e0 con 0x7fb6e8137780 2024-04-19T01:42:27.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ef95a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6e809c000 con 0x7fb6e813cc60 2024-04-19T01:42:27.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ef95a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ef95a640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fb6e809bd20 con 0x7fb6e80a60b0 2024-04-19T01:42:27.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ef95a640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:27.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ef95a640 10 monclient(hunting): authenticate will time out at 933.001881s 2024-04-19T01:42:27.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ed6f6640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fb6e8137780 0x7fb6e8137b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:33588/0 (socket says 172.21.15.62:33588) 2024-04-19T01:42:27.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ed6f6640 1 -- 172.21.15.62:0/2693906042 learned_addr learned my addr 172.21.15.62:0/2693906042 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:42:27.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ecef5640 1 --2- 172.21.15.62:0/2693906042 >> v2:172.21.15.62:3300/0 conn(0x7fb6e80a60b0 0x7fb6e813c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ecef5640 10 monclient(hunting): get_auth_request con 0x7fb6e80a60b0 auth_method 0 2024-04-19T01:42:27.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ecef5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:27.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ecef5640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:27.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ecef5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:27.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ecef5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:27.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ecef5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:27.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ecef5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:27.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ecef5640 10 monclient(hunting): handle_auth_done global_id 4278 payload 274 2024-04-19T01:42:27.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ecef5640 1 -- 172.21.15.62:0/2693906042 >> v1:172.21.15.71:6789/0 conn(0x7fb6e813cc60 legacy=0x7fb6e813f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:27.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ecef5640 1 -- 172.21.15.62:0/2693906042 >> v1:172.21.15.62:6789/0 conn(0x7fb6e8137780 legacy=0x7fb6e8137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:27.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ecef5640 10 monclient: _finish_hunting 0 2024-04-19T01:42:27.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ecef5640 1 monclient: found mon.noname-a 2024-04-19T01:42:27.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ecef5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:27.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ecef5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:27.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.178+0000 7fb6ecef5640 1 -- 172.21.15.62:0/2693906042 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6e8138270 con 0x7fb6e80a60b0 2024-04-19T01:42:27.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.182+0000 7fb6ecef5640 1 -- 172.21.15.62:0/2693906042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6e80a60b0 msgr2=0x7fb6e813c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:27.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.182+0000 7fb6ecef5640 1 --2- 172.21.15.62:0/2693906042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6e80a60b0 0x7fb6e813c720 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb6d80097c0 tx=0x7fb6d801c210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b97ac728f16a584c server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:27.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6dffff640 1 -- 172.21.15.62:0/2693906042 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d80b0070 con 0x7fb6e80a60b0 2024-04-19T01:42:27.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:27.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6dffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:27.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6dffff640 10 monclient: dump: 2024-04-19T01:42:27.186 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:27.186 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.186 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.186 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.186 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:27.186 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.186 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:27.187 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:27.187 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:27.187 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6dffff640 10 monclient: _finish_auth 0 2024-04-19T01:42:27.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6dffff640 10 monclient: _check_auth_tickets 2024-04-19T01:42:27.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:27.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6dffff640 1 -- 172.21.15.62:0/2693906042 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6d80ab600 con 0x7fb6e80a60b0 2024-04-19T01:42:27.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6dffff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:27.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6dffff640 1 -- 172.21.15.62:0/2693906042 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d80ab9d0 con 0x7fb6e80a60b0 2024-04-19T01:42:27.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:27.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6dffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:27.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6dffff640 10 monclient: dump: 2024-04-19T01:42:27.187 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:27.187 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.188 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.188 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.188 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:27.188 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.188 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:27.188 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:27.188 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:27.188 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 5 monclient: authenticate success, global_id 4278 2024-04-19T01:42:27.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:27.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 4 set_mon_vals no callback set 2024-04-19T01:42:27.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 10 monclient: shutdown 2024-04-19T01:42:27.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:27.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 1 -- 172.21.15.62:0/2693906042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6e80a60b0 msgr2=0x7fb6e813c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 1 --2- 172.21.15.62:0/2693906042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6e80a60b0 0x7fb6e813c720 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb6d80097c0 tx=0x7fb6d801c210 comp rx=0 tx=0).stop 2024-04-19T01:42:27.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 1 -- 172.21.15.62:0/2693906042 shutdown_connections 2024-04-19T01:42:27.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 1 --2- 172.21.15.62:0/2693906042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6e80a60b0 0x7fb6e813c720 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 1 -- 172.21.15.62:0/2693906042 >> 172.21.15.62:0/2693906042 conn(0x7fb6e80aa1d0 msgr2=0x7fb6e80a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:27.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 1 -- 172.21.15.62:0/2693906042 shutdown_connections 2024-04-19T01:42:27.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 1 -- 172.21.15.62:0/2693906042 wait complete. 2024-04-19T01:42:27.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 10 monclient: build_initial_monmap 2024-04-19T01:42:27.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 10 monclient: monmap: 2024-04-19T01:42:27.192 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:27.192 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:27.192 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:27.192 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:27.192 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:27.192 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.192 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:27.192 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:27.192 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:27.192 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 1 Processor -- start 2024-04-19T01:42:27.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 1 -- start start 2024-04-19T01:42:27.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 10 monclient: init 2024-04-19T01:42:27.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:27.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-19T01:42:27.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6e8137780 0x7fb6e81cf630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 10 monclient(hunting): picked mon.noname-a con 0x7fb6e8137780 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:27.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb6e813cc60 0x7fb6e81364c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 10 monclient(hunting): picked mon.noname-c con 0x7fb6e813cc60 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:27.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6e809e9d0 0x7fb6e809ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 10 monclient(hunting): picked mon.noname-b con 0x7fb6e809e9d0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:27.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e8143320 con 0x7fb6e8137780 2024-04-19T01:42:27.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e81431a0 con 0x7fb6e809e9d0 2024-04-19T01:42:27.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fb6e81434a0 con 0x7fb6e813cc60 2024-04-19T01:42:27.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:27.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ef95a640 10 monclient(hunting): authenticate will time out at 933.011842s 2024-04-19T01:42:27.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.186+0000 7fb6ecef5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb6e813cc60 0x7fb6e81364c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ecef5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb6e813cc60 0x7fb6e81364c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60440/0 (socket says 172.21.15.62:60440) 2024-04-19T01:42:27.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ecef5640 1 -- 172.21.15.62:0/633346152 learned_addr learned my addr 172.21.15.62:0/633346152 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:27.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ecef5640 10 monclient(hunting): get_auth_request con 0x7fb6e813cc60 auth_method 0 2024-04-19T01:42:27.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ecef5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:27.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ecef5640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:27.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ecef5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:27.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ecef5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:27.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ecef5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:27.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ecef5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:27.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6edef7640 1 --2- 172.21.15.62:0/633346152 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6e809e9d0 0x7fb6e809ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6edef7640 10 monclient(hunting): get_auth_request con 0x7fb6e809e9d0 auth_method 0 2024-04-19T01:42:27.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6edef7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:27.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6edef7640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:27.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6edef7640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:27.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ed6f6640 1 --2- 172.21.15.62:0/633346152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6e8137780 0x7fb6e81cf630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ecef5640 10 monclient(hunting): handle_auth_done global_id 4283 payload 824 2024-04-19T01:42:27.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ecef5640 1 -- 172.21.15.62:0/633346152 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6e809e9d0 msgr2=0x7fb6e809ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ecef5640 1 --2- 172.21.15.62:0/633346152 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6e809e9d0 0x7fb6e809ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ecef5640 1 -- 172.21.15.62:0/633346152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6e8137780 msgr2=0x7fb6e81cf630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ecef5640 1 --2- 172.21.15.62:0/633346152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6e8137780 0x7fb6e81cf630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ecef5640 10 monclient: _finish_hunting 0 2024-04-19T01:42:27.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ecef5640 1 monclient: found mon.noname-c 2024-04-19T01:42:27.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ecef5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:27.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ecef5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:42:27.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ecef5640 1 -- 172.21.15.62:0/633346152 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6e80a0670 con 0x7fb6e813cc60 2024-04-19T01:42:27.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6edef7640 1 --2- 172.21.15.62:0/633346152 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6e809e9d0 0x7fb6e809ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T01:42:27.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ed6f6640 10 monclient: get_auth_request con 0x7fb6e8137780 auth_method 0 2024-04-19T01:42:27.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ed6f6640 1 --2- 172.21.15.62:0/633346152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6e8137780 0x7fb6e81cf630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:42:27.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ecef5640 1 -- 172.21.15.62:0/633346152 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb6e813cc60 msgr2=0x7fb6e81364c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:27.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ecef5640 1 --2- 172.21.15.62:0/633346152 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb6e813cc60 0x7fb6e81364c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb6d80097c0 tx=0x7fb6d8018fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:27.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ddffb640 1 -- 172.21.15.62:0/633346152 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d80b0040 con 0x7fb6e813cc60 2024-04-19T01:42:27.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:27.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ddffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:27.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ddffb640 10 monclient: dump: 2024-04-19T01:42:27.198 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:27.198 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.198 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.198 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.198 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:27.198 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.198 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:27.198 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:27.198 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:27.198 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ddffb640 10 monclient: _finish_auth 0 2024-04-19T01:42:27.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ddffb640 10 monclient: _check_auth_tickets 2024-04-19T01:42:27.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:27.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ef95a640 5 monclient: authenticate success, global_id 4283 2024-04-19T01:42:27.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ef95a640 10 monclient: _renew_subs 2024-04-19T01:42:27.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ef95a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:27.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ef95a640 1 -- 172.21.15.62:0/633346152 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6e81d0520 con 0x7fb6e813cc60 2024-04-19T01:42:27.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ef95a640 10 monclient: _renew_subs 2024-04-19T01:42:27.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ef95a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:27.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ef95a640 1 -- 172.21.15.62:0/633346152 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6e8136a00 con 0x7fb6e813cc60 2024-04-19T01:42:27.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ddffb640 1 -- 172.21.15.62:0/633346152 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6d8004030 con 0x7fb6e813cc60 2024-04-19T01:42:27.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ddffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:27.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ddffb640 1 -- 172.21.15.62:0/633346152 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d80043e0 con 0x7fb6e813cc60 2024-04-19T01:42:27.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:27.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:27.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6ddffb640 10 monclient: dump: 2024-04-19T01:42:27.199 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:27.199 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.199 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.200 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.200 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:27.200 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.200 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:27.200 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:27.200 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:27.200 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.190+0000 7fb6df7fe640 4 set_mon_vals no callback set 2024-04-19T01:42:27.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.194+0000 7fb6d37fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:27.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.194+0000 7fb6d37fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:27.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.194+0000 7fb6ddffb640 1 -- 172.21.15.62:0/633346152 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fb6d80b3070 con 0x7fb6e813cc60 2024-04-19T01:42:27.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.194+0000 7fb6d37fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:27.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.194+0000 7fb6ddffb640 1 --2- 172.21.15.62:0/633346152 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fb6cc036420 0x7fb6cc0388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.194+0000 7fb6ddffb640 1 -- 172.21.15.62:0/633346152 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4346+0+0 (secure 0 0 0) 0x7fb6d80e3240 con 0x7fb6e813cc60 2024-04-19T01:42:27.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.194+0000 7fb6d37fe640 1 -- 172.21.15.62:0/633346152 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6c0005090 con 0x7fb6e813cc60 2024-04-19T01:42:27.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.194+0000 7fb6ed6f6640 1 --2- 172.21.15.62:0/633346152 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fb6cc036420 0x7fb6cc0388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.194+0000 7fb6ed6f6640 10 monclient: get_auth_request con 0x7fb6cc036420 auth_method 0 2024-04-19T01:42:27.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.198+0000 7fb6ed6f6640 1 --2- 172.21.15.62:0/633346152 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fb6cc036420 0x7fb6cc0388e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb6e4007fd0 tx=0x7fb6e4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:27.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.198+0000 7f505ffff640 10 monclient: get_monmap_and_config 2024-04-19T01:42:27.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.198+0000 7f505ffff640 10 monclient: build_initial_monmap 2024-04-19T01:42:27.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.198+0000 7f505ffff640 10 monclient: monmap: 2024-04-19T01:42:27.202 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:27.202 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.202 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:27.203425+0000 2024-04-19T01:42:27.202 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:27.203425+0000 2024-04-19T01:42:27.202 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:27.202 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.202 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:27.202 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:27.202 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:27.202 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:27.202 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:27.203 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.198+0000 7fb6ddffb640 1 -- 172.21.15.62:0/633346152 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb6d80b9020 con 0x7fb6e813cc60 2024-04-19T01:42:27.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.198+0000 7fb6ddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:27.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.198+0000 7fb6ddffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:27.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.198+0000 7f505ffff640 1 Processor -- start 2024-04-19T01:42:27.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.198+0000 7f505ffff640 1 -- start start 2024-04-19T01:42:27.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.198+0000 7f505ffff640 10 monclient: init 2024-04-19T01:42:27.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.198+0000 7f505ffff640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:27.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.198+0000 7f505ffff640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-04-19T01:42:27.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.198+0000 7f505ffff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5058027780 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:27.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.198+0000 7f505ffff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5058027b60 0x7f5058027f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.198+0000 7f505ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f5058027b60 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:27.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.198+0000 7f505ffff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f50580c10e0 0x7f50580c34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.198+0000 7f505ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f50580c10e0 addr v2:172.21.15.71:3300/0 2024-04-19T01:42:27.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.198+0000 7f505ffff640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5058024610 con 0x7f5058027780 2024-04-19T01:42:27.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.202+0000 7f505ffff640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.202+0000 7f505ffff640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f5058024eb0 con 0x7f5058027b60 2024-04-19T01:42:27.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.202+0000 7f505ffff640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.202+0000 7f505ffff640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f50580248f0 con 0x7f50580c10e0 2024-04-19T01:42:27.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.202+0000 7f505ffff640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:27.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.202+0000 7f505ffff640 10 monclient(hunting): authenticate will time out at 933.024037s 2024-04-19T01:42:27.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.202+0000 7f505e7fc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5058027b60 0x7f5058027f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.202+0000 7f505e7fc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5058027b60 0x7f5058027f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36378/0 (socket says 172.21.15.62:36378) 2024-04-19T01:42:27.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.202+0000 7f505e7fc640 1 -- 172.21.15.62:0/1658751913 learned_addr learned my addr 172.21.15.62:0/1658751913 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:27.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.202+0000 7f505e7fc640 10 monclient(hunting): get_auth_request con 0x7f5058027b60 auth_method 0 2024-04-19T01:42:27.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.202+0000 7f505e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:27.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.202+0000 7f505e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:27.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.202+0000 7f505e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:27.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.202+0000 7f505e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:27.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.202+0000 7f505e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:27.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.202+0000 7f505e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:27.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.202+0000 7f505f7fe640 1 --2- 172.21.15.62:0/1658751913 >> v2:172.21.15.71:3300/0 conn(0x7f50580c10e0 0x7f50580c34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.202+0000 7f505e7fc640 10 monclient(hunting): handle_auth_done global_id 4281 payload 274 2024-04-19T01:42:27.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.202+0000 7f505e7fc640 1 -- 172.21.15.62:0/1658751913 >> v2:172.21.15.71:3300/0 conn(0x7f50580c10e0 msgr2=0x7f50580c34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:27.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.202+0000 7f505e7fc640 1 --2- 172.21.15.62:0/1658751913 >> v2:172.21.15.71:3300/0 conn(0x7f50580c10e0 0x7f50580c34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.202+0000 7f505e7fc640 1 -- 172.21.15.62:0/1658751913 >> v1:172.21.15.62:6789/0 conn(0x7f5058027780 legacy=0x7f50580b8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:27.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.202+0000 7f505e7fc640 10 monclient: _finish_hunting 0 2024-04-19T01:42:27.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.202+0000 7f505e7fc640 1 monclient: found mon.noname-a 2024-04-19T01:42:27.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.202+0000 7f505e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:27.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.202+0000 7f505e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:27.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.202+0000 7f505e7fc640 1 -- 172.21.15.62:0/1658751913 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5058029c50 con 0x7f5058027b60 2024-04-19T01:42:27.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505e7fc640 1 -- 172.21.15.62:0/1658751913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5058027b60 msgr2=0x7f5058027f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:27.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505e7fc640 1 --2- 172.21.15.62:0/1658751913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5058027b60 0x7f5058027f40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f50580285d0 tx=0x7f505401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=716871a53a5a0544 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:27.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505dffb640 1 -- 172.21.15.62:0/1658751913 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50540b0070 con 0x7f5058027b60 2024-04-19T01:42:27.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:27.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:27.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505dffb640 10 monclient: dump: 2024-04-19T01:42:27.221 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:27.221 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.221 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.221 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.221 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:27.221 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.221 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:27.221 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:27.221 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:27.221 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505dffb640 10 monclient: _finish_auth 0 2024-04-19T01:42:27.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505dffb640 10 monclient: _check_auth_tickets 2024-04-19T01:42:27.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:27.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505dffb640 1 -- 172.21.15.62:0/1658751913 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50540a97f0 con 0x7f5058027b60 2024-04-19T01:42:27.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505dffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:27.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505dffb640 1 -- 172.21.15.62:0/1658751913 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50540a9bc0 con 0x7f5058027b60 2024-04-19T01:42:27.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:27.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:27.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505dffb640 10 monclient: dump: 2024-04-19T01:42:27.222 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:27.222 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.222 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.222 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.222 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:27.222 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.222 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:27.222 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:27.223 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:27.223 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 5 monclient: authenticate success, global_id 4281 2024-04-19T01:42:27.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:27.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 4 set_mon_vals no callback set 2024-04-19T01:42:27.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 10 monclient: shutdown 2024-04-19T01:42:27.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:27.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 1 -- 172.21.15.62:0/1658751913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5058027b60 msgr2=0x7f5058027f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 1 --2- 172.21.15.62:0/1658751913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5058027b60 0x7f5058027f40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f50580285d0 tx=0x7f505401c3e0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 1 -- 172.21.15.62:0/1658751913 shutdown_connections 2024-04-19T01:42:27.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 1 --2- 172.21.15.62:0/1658751913 >> v2:172.21.15.71:3300/0 conn(0x7f50580c10e0 0x7f50580c34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 1 --2- 172.21.15.62:0/1658751913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5058027b60 0x7f5058027f40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 1 -- 172.21.15.62:0/1658751913 >> 172.21.15.62:0/1658751913 conn(0x7f5058006800 msgr2=0x7f5058006c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:27.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 1 -- 172.21.15.62:0/1658751913 shutdown_connections 2024-04-19T01:42:27.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 1 -- 172.21.15.62:0/1658751913 wait complete. 2024-04-19T01:42:27.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 10 monclient: build_initial_monmap 2024-04-19T01:42:27.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 10 monclient: monmap: 2024-04-19T01:42:27.224 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:27.224 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:27.224 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:27.224 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:27.224 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:27.224 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.224 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:27.224 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:27.224 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:27.224 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 1 Processor -- start 2024-04-19T01:42:27.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 1 -- start start 2024-04-19T01:42:27.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 10 monclient: init 2024-04-19T01:42:27.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:27.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-19T01:42:27.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5058027780 0x7f5058153b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f5058027780 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:27.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5058027b60 0x7f50580c00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f5058027b60 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:27.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f50580c10e0 0x7f50580c0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f50580c10e0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:27.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f50580c65c0 con 0x7f5058027780 2024-04-19T01:42:27.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f50580c6440 con 0x7f5058027b60 2024-04-19T01:42:27.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f50580c6740 con 0x7f50580c10e0 2024-04-19T01:42:27.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:27.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505ffff640 10 monclient(hunting): authenticate will time out at 933.031432s 2024-04-19T01:42:27.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505effd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5058027780 0x7f5058153b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505effd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5058027780 0x7f5058153b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36384/0 (socket says 172.21.15.62:36384) 2024-04-19T01:42:27.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505effd640 1 -- 172.21.15.62:0/231798331 learned_addr learned my addr 172.21.15.62:0/231798331 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:27.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505effd640 10 monclient(hunting): get_auth_request con 0x7f5058027780 auth_method 0 2024-04-19T01:42:27.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:27.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505effd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:27.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:27.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505f7fe640 1 --2- 172.21.15.62:0/231798331 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f50580c10e0 0x7f50580c0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505f7fe640 10 monclient(hunting): get_auth_request con 0x7f50580c10e0 auth_method 0 2024-04-19T01:42:27.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:27.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:27.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:27.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:27.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:27.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:27.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:27.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:27.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505e7fc640 1 --2- 172.21.15.62:0/231798331 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5058027b60 0x7f50580c00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:27.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505f7fe640 10 monclient(hunting): handle_auth_done global_id 4286 payload 824 2024-04-19T01:42:27.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505f7fe640 1 -- 172.21.15.62:0/231798331 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5058027b60 msgr2=0x7f50580c00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505f7fe640 1 --2- 172.21.15.62:0/231798331 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5058027b60 0x7f50580c00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505f7fe640 1 -- 172.21.15.62:0/231798331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5058027780 msgr2=0x7f5058153b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505f7fe640 1 --2- 172.21.15.62:0/231798331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5058027780 0x7f5058153b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505f7fe640 10 monclient: _finish_hunting 0 2024-04-19T01:42:27.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505f7fe640 1 monclient: found mon.noname-c 2024-04-19T01:42:27.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:27.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:42:27.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.206+0000 7f505f7fe640 1 -- 172.21.15.62:0/231798331 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f505802cdd0 con 0x7f50580c10e0 2024-04-19T01:42:27.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.210+0000 7f505f7fe640 1 -- 172.21.15.62:0/231798331 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f50580c10e0 msgr2=0x7f50580c0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:27.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.210+0000 7f505f7fe640 1 --2- 172.21.15.62:0/231798331 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f50580c10e0 0x7f50580c0620 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5060069920 tx=0x7f5060063510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:27.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.210+0000 7f50477fe640 1 -- 172.21.15.62:0/231798331 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5060095c30 con 0x7f50580c10e0 2024-04-19T01:42:27.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.210+0000 7f50477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:27.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.210+0000 7f50477fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:27.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.210+0000 7f50477fe640 10 monclient: dump: 2024-04-19T01:42:27.228 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:27.228 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.228 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.228 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.228 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:27.228 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.228 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:27.228 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:27.228 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:27.229 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.210+0000 7f50477fe640 10 monclient: _finish_auth 0 2024-04-19T01:42:27.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.210+0000 7f50477fe640 10 monclient: _check_auth_tickets 2024-04-19T01:42:27.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.210+0000 7f50477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:27.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.210+0000 7f50477fe640 1 -- 172.21.15.62:0/231798331 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5060095df0 con 0x7f50580c10e0 2024-04-19T01:42:27.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.210+0000 7f505ffff640 5 monclient: authenticate success, global_id 4286 2024-04-19T01:42:27.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.210+0000 7f505ffff640 10 monclient: _renew_subs 2024-04-19T01:42:27.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.210+0000 7f505ffff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:27.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.210+0000 7f505ffff640 1 -- 172.21.15.62:0/231798331 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5058023ac0 con 0x7f50580c10e0 2024-04-19T01:42:27.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.210+0000 7f50477fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:27.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.210+0000 7f50477fe640 1 -- 172.21.15.62:0/231798331 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f506006ab80 con 0x7f50580c10e0 2024-04-19T01:42:27.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.210+0000 7f50477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:27.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.210+0000 7f50477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:27.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.210+0000 7f50477fe640 10 monclient: dump: 2024-04-19T01:42:27.229 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:27.229 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.230 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.230 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.230 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:27.230 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.230 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:27.230 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:27.230 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:27.230 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.210+0000 7f505d7fa640 4 set_mon_vals no callback set 2024-04-19T01:42:27.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.210+0000 7f505ffff640 10 monclient: _renew_subs 2024-04-19T01:42:27.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.210+0000 7f505ffff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:27.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.210+0000 7f505ffff640 1 -- 172.21.15.62:0/231798331 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f50581547c0 con 0x7f50580c10e0 2024-04-19T01:42:27.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.214+0000 7f50477fe640 1 -- 172.21.15.62:0/231798331 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f506006adc0 con 0x7f50580c10e0 2024-04-19T01:42:27.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.214+0000 7f505ffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:27.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.214+0000 7f50477fe640 1 --2- 172.21.15.62:0/231798331 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f504c036470 0x7f504c038930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.214+0000 7f50477fe640 1 -- 172.21.15.62:0/231798331 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4346+0+0 (secure 0 0 0) 0x7f50600cd5b0 con 0x7f50580c10e0 2024-04-19T01:42:27.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.214+0000 7f505ffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:27.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.214+0000 7f505ffff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:27.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.214+0000 7f505ffff640 1 -- 172.21.15.62:0/231798331 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5058005720 con 0x7f50580c10e0 2024-04-19T01:42:27.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.214+0000 7f505effd640 1 --2- 172.21.15.62:0/231798331 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f504c036470 0x7f504c038930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.214+0000 7f505effd640 10 monclient: get_auth_request con 0x7f504c036470 auth_method 0 2024-04-19T01:42:27.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.214+0000 7f505effd640 1 --2- 172.21.15.62:0/231798331 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f504c036470 0x7f504c038930 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f50500050b0 tx=0x7f5050003700 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:27.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.222+0000 7f50477fe640 1 -- 172.21.15.62:0/231798331 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f50600fee70 con 0x7f50580c10e0 2024-04-19T01:42:27.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.222+0000 7f50477fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:27.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.222+0000 7f50477fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:27.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.298+0000 7f06a8205640 10 monclient: get_monmap_and_config 2024-04-19T01:42:27.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.298+0000 7f06a8205640 10 monclient: build_initial_monmap 2024-04-19T01:42:27.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.298+0000 7f06a8205640 10 monclient: monmap: 2024-04-19T01:42:27.300 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:27.300 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.300 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:27.303869+0000 2024-04-19T01:42:27.300 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:27.303869+0000 2024-04-19T01:42:27.300 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:27.300 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.300 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:27.300 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:27.300 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:27.300 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:27.300 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:27.301 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.298+0000 7f06a8205640 1 Processor -- start 2024-04-19T01:42:27.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.298+0000 7f06a8205640 1 -- start start 2024-04-19T01:42:27.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.298+0000 7f06a8205640 10 monclient: init 2024-04-19T01:42:27.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.298+0000 7f06a8205640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:27.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.298+0000 7f06a8205640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-04-19T01:42:27.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.298+0000 7f06a8205640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f06a0137780 0x7f06a0137b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.298+0000 7f06a8205640 10 monclient(hunting): picked mon.noname-a con 0x7f06a0137780 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:27.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.298+0000 7f06a8205640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f06a01380a0 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:27.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.298+0000 7f06a8205640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06a00a60b0 0x7f06a00a4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.298+0000 7f06a8205640 10 monclient(hunting): picked mon.noname-b con 0x7f06a00a60b0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:27.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.298+0000 7f06a67a2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06a00a60b0 0x7f06a00a4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.298+0000 7f06a67a2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06a00a60b0 0x7f06a00a4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60466/0 (socket says 172.21.15.62:60466) 2024-04-19T01:42:27.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.298+0000 7f06a67a2640 1 -- 172.21.15.62:0/1006498354 learned_addr learned my addr 172.21.15.62:0/1006498354 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:27.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.298+0000 7f06a8205640 1 -- 172.21.15.62:0/1006498354 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06a009c2e0 con 0x7f06a01380a0 2024-04-19T01:42:27.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.298+0000 7f06a5fa1640 1 --2- 172.21.15.62:0/1006498354 >> v2:172.21.15.62:3300/0 conn(0x7f06a0137780 0x7f06a0137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.298+0000 7f06a8205640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.302+0000 7f06a8205640 1 -- 172.21.15.62:0/1006498354 --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f06a009bd20 con 0x7f06a0137780 2024-04-19T01:42:27.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.302+0000 7f06a8205640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.302+0000 7f06a8205640 1 -- 172.21.15.62:0/1006498354 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f06a009c000 con 0x7f06a00a60b0 2024-04-19T01:42:27.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.302+0000 7f06a8205640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:27.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.302+0000 7f06a8205640 10 monclient(hunting): authenticate will time out at 933.124336s 2024-04-19T01:42:27.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.302+0000 7f06a67a2640 10 monclient(hunting): get_auth_request con 0x7f06a00a60b0 auth_method 0 2024-04-19T01:42:27.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.302+0000 7f06a67a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:27.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.302+0000 7f06a67a2640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.302+0000 7f06a67a2640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.302+0000 7f06a5fa1640 10 monclient(hunting): get_auth_request con 0x7f06a0137780 auth_method 0 2024-04-19T01:42:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.302+0000 7f06a5fa1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.302+0000 7f06a5fa1640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.302+0000 7f06a5fa1640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.302+0000 7f06a5fa1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.302+0000 7f06a5fa1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.302+0000 7f06a5fa1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.302+0000 7f06a67a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.302+0000 7f06a67a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.302+0000 7f06a67a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:27.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.302+0000 7f06a67a2640 10 monclient(hunting): handle_auth_done global_id 4289 payload 274 2024-04-19T01:42:27.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.302+0000 7f06a67a2640 1 -- 172.21.15.62:0/1006498354 >> v2:172.21.15.62:3300/0 conn(0x7f06a0137780 msgr2=0x7f06a0137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:27.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.302+0000 7f06a67a2640 1 --2- 172.21.15.62:0/1006498354 >> v2:172.21.15.62:3300/0 conn(0x7f06a0137780 0x7f06a0137b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.302+0000 7f06a67a2640 1 -- 172.21.15.62:0/1006498354 >> v1:172.21.15.62:6789/0 conn(0x7f06a01380a0 legacy=0x7f06a00a45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:27.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.302+0000 7f06a67a2640 10 monclient: _finish_hunting 0 2024-04-19T01:42:27.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.302+0000 7f06a67a2640 1 monclient: found mon.noname-b 2024-04-19T01:42:27.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.302+0000 7f06a67a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:27.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.302+0000 7f06a67a2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:42:27.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.302+0000 7f06a67a2640 1 -- 172.21.15.62:0/1006498354 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06a0073210 con 0x7f06a00a60b0 2024-04-19T01:42:27.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.306+0000 7f06a67a2640 1 -- 172.21.15.62:0/1006498354 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06a00a60b0 msgr2=0x7f06a00a4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:27.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.306+0000 7f06a67a2640 1 --2- 172.21.15.62:0/1006498354 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06a00a60b0 0x7f06a00a4ce0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0698009e50 tx=0x7f069801c710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d98e612cbefa119 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:27.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.306+0000 7f06a4f9f640 1 -- 172.21.15.62:0/1006498354 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06980ac020 con 0x7f06a00a60b0 2024-04-19T01:42:27.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.306+0000 7f06a4f9f640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:27.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.306+0000 7f06a4f9f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:42:27.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.306+0000 7f06a4f9f640 10 monclient: dump: 2024-04-19T01:42:27.310 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:27.310 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.310 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.310 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.310 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:27.310 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.310 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:27.311 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:27.311 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:27.311 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.306+0000 7f06a4f9f640 10 monclient: _finish_auth 0 2024-04-19T01:42:27.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.306+0000 7f06a4f9f640 10 monclient: _check_auth_tickets 2024-04-19T01:42:27.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.306+0000 7f06a4f9f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:27.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.306+0000 7f06a4f9f640 1 -- 172.21.15.62:0/1006498354 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06980a6470 con 0x7f06a00a60b0 2024-04-19T01:42:27.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.306+0000 7f06a8205640 5 monclient: authenticate success, global_id 4289 2024-04-19T01:42:27.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.306+0000 7f06a8205640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:42:27.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.306+0000 7f06a4f9f640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:27.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.306+0000 7f06a4f9f640 1 -- 172.21.15.62:0/1006498354 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06980a6840 con 0x7f06a00a60b0 2024-04-19T01:42:27.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.306+0000 7f06a4f9f640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:27.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.306+0000 7f06a4f9f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:27.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.306+0000 7f06a4f9f640 10 monclient: dump: 2024-04-19T01:42:27.312 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:27.312 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.312 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.312 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.312 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:27.312 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.312 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:27.312 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:27.312 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:27.312 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.306+0000 7f06a8205640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:27.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.306+0000 7f06a8205640 4 set_mon_vals no callback set 2024-04-19T01:42:27.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.306+0000 7f06a8205640 10 monclient: shutdown 2024-04-19T01:42:27.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.306+0000 7f06a8205640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:27.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.306+0000 7f06a8205640 1 -- 172.21.15.62:0/1006498354 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06a00a60b0 msgr2=0x7f06a00a4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.306+0000 7f06a8205640 1 --2- 172.21.15.62:0/1006498354 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06a00a60b0 0x7f06a00a4ce0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0698009e50 tx=0x7f069801c710 comp rx=0 tx=0).stop 2024-04-19T01:42:27.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a8205640 1 -- 172.21.15.62:0/1006498354 shutdown_connections 2024-04-19T01:42:27.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a8205640 1 --2- 172.21.15.62:0/1006498354 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06a00a60b0 0x7f06a00a4ce0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a8205640 1 --2- 172.21.15.62:0/1006498354 >> v2:172.21.15.62:3300/0 conn(0x7f06a0137780 0x7f06a0137b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a8205640 1 -- 172.21.15.62:0/1006498354 >> 172.21.15.62:0/1006498354 conn(0x7f06a00aa1d0 msgr2=0x7f06a00a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:27.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a8205640 1 -- 172.21.15.62:0/1006498354 shutdown_connections 2024-04-19T01:42:27.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a8205640 1 -- 172.21.15.62:0/1006498354 wait complete. 2024-04-19T01:42:27.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a8205640 10 monclient: build_initial_monmap 2024-04-19T01:42:27.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a8205640 10 monclient: monmap: 2024-04-19T01:42:27.315 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:27.315 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:27.315 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:27.315 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:27.315 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:27.315 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.315 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:27.316 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:27.316 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:27.316 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a8205640 1 Processor -- start 2024-04-19T01:42:27.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a8205640 1 -- start start 2024-04-19T01:42:27.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a8205640 10 monclient: init 2024-04-19T01:42:27.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a8205640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:27.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a8205640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-19T01:42:27.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a8205640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06a00a60b0 0x7f06a01cf5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a8205640 10 monclient(hunting): picked mon.noname-c con 0x7f06a00a60b0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:27.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a8205640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f06a0137780 0x7f06a01cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a8205640 10 monclient(hunting): picked mon.noname-b con 0x7f06a0137780 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:27.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a8205640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06a01380a0 0x7f06a013c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7fa7d81ea640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-19T01:42:27.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7fa7d81ea640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-19T01:42:27.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7fa7d81ea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:27.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7fa7d81ea640 1 -- 172.21.15.62:0/3835028680 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fa7d013f520 con 0x7fa7d01380a0 2024-04-19T01:42:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a5fa1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06a00a60b0 0x7f06a01cf5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a5fa1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06a00a60b0 0x7f06a01cf5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60470/0 (socket says 172.21.15.62:60470) 2024-04-19T01:42:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a67a2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06a01380a0 0x7f06a013c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a67a2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06a01380a0 0x7f06a013c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36398/0 (socket says 172.21.15.62:36398) 2024-04-19T01:42:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a5fa1640 1 -- 172.21.15.62:0/1709664659 learned_addr learned my addr 172.21.15.62:0/1709664659 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a8205640 10 monclient(hunting): picked mon.noname-a con 0x7f06a01380a0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a8205640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a8205640 1 -- 172.21.15.62:0/1709664659 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f06a0142170 con 0x7f06a01380a0 2024-04-19T01:42:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a8205640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a8205640 1 -- 172.21.15.62:0/1709664659 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f06a00a5430 con 0x7f06a0137780 2024-04-19T01:42:27.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a8205640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a8205640 1 -- 172.21.15.62:0/1709664659 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f06a01422f0 con 0x7f06a00a60b0 2024-04-19T01:42:27.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a8205640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:27.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a8205640 10 monclient(hunting): authenticate will time out at 933.135705s 2024-04-19T01:42:27.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7fa7c67fc640 1 -- 172.21.15.62:0/3835028680 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa7c801a540 con 0x7fa7d01380a0 2024-04-19T01:42:27.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7fa7c67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-19T01:42:27.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7fa7c67fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:27.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a57a0640 1 --2- 172.21.15.62:0/1709664659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f06a0137780 0x7f06a01cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a5fa1640 10 monclient(hunting): get_auth_request con 0x7f06a00a60b0 auth_method 0 2024-04-19T01:42:27.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a5fa1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:27.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a5fa1640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:27.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a5fa1640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:27.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a57a0640 10 monclient(hunting): get_auth_request con 0x7f06a0137780 auth_method 0 2024-04-19T01:42:27.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a57a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:27.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a57a0640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:27.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a57a0640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:27.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a5fa1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:27.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a5fa1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:27.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a5fa1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:27.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a67a2640 10 monclient(hunting): get_auth_request con 0x7f06a01380a0 auth_method 0 2024-04-19T01:42:27.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a67a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:27.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a67a2640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:27.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.310+0000 7f06a67a2640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:27.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06a57a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:27.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06a57a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:27.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06a57a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:27.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06a67a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:27.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06a67a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:27.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06a67a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:27.322 INFO:teuthology.orchestra.run.smithi062.stdout:47244640259 2024-04-19T01:42:27.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06a5fa1640 10 monclient(hunting): handle_auth_done global_id 4292 payload 824 2024-04-19T01:42:27.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06a5fa1640 1 -- 172.21.15.62:0/1709664659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f06a0137780 msgr2=0x7f06a01cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06a5fa1640 1 --2- 172.21.15.62:0/1709664659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f06a0137780 0x7f06a01cfd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06a5fa1640 1 -- 172.21.15.62:0/1709664659 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06a01380a0 msgr2=0x7f06a013c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06a5fa1640 1 --2- 172.21.15.62:0/1709664659 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06a01380a0 0x7f06a013c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06a5fa1640 10 monclient: _finish_hunting 0 2024-04-19T01:42:27.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06a5fa1640 1 monclient: found mon.noname-c 2024-04-19T01:42:27.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06a5fa1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:27.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06a5fa1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:42:27.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06a5fa1640 1 -- 172.21.15.62:0/1709664659 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06a00a2150 con 0x7f06a00a60b0 2024-04-19T01:42:27.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7fa7d81ea640 1 -- 172.21.15.62:0/3835028680 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa7b4036450 msgr2=0x7fa7b4038910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7fa7d81ea640 1 --2- 172.21.15.62:0/3835028680 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa7b4036450 0x7fa7b4038910 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa7d013db70 tx=0x7fa7cc008040 comp rx=0 tx=0).stop 2024-04-19T01:42:27.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7fa7d81ea640 10 monclient: shutdown 2024-04-19T01:42:27.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7fa7d81ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:27.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7fa7d81ea640 1 -- 172.21.15.62:0/3835028680 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7d01380a0 msgr2=0x7fa7d013c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7fa7d81ea640 1 --2- 172.21.15.62:0/3835028680 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7d01380a0 0x7fa7d013c810 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa7c800caf0 tx=0x7fa7c800cef0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7fa7d81ea640 1 -- 172.21.15.62:0/3835028680 shutdown_connections 2024-04-19T01:42:27.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7fa7d81ea640 1 --2- 172.21.15.62:0/3835028680 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa7d01380a0 0x7fa7d013c810 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7fa7d81ea640 1 --2- 172.21.15.62:0/3835028680 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa7d0137780 0x7fa7d013c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7fa7d81ea640 1 --2- 172.21.15.62:0/3835028680 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa7b4036450 0x7fa7b4038910 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7fa7d81ea640 1 --2- 172.21.15.62:0/3835028680 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa7d00a60b0 0x7fa7d01364d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7fa7d81ea640 1 -- 172.21.15.62:0/3835028680 >> 172.21.15.62:0/3835028680 conn(0x7fa7d00aa1d0 msgr2=0x7fa7d00a79d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:27.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7fa7d81ea640 1 -- 172.21.15.62:0/3835028680 shutdown_connections 2024-04-19T01:42:27.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7fa7d81ea640 1 -- 172.21.15.62:0/3835028680 wait complete. 2024-04-19T01:42:27.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06a5fa1640 1 -- 172.21.15.62:0/1709664659 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06a00a60b0 msgr2=0x7f06a01cf5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:27.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06a5fa1640 1 --2- 172.21.15.62:0/1709664659 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06a00a60b0 0x7f06a01cf5a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f069c00e8b0 tx=0x7f069c00ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:27.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06967fc640 1 -- 172.21.15.62:0/1709664659 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069c017070 con 0x7f06a00a60b0 2024-04-19T01:42:27.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:27.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06967fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:27.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06967fc640 10 monclient: dump: 2024-04-19T01:42:27.328 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:27.328 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.328 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.328 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.328 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:27.328 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.328 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:27.328 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:27.328 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:27.328 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06967fc640 10 monclient: _finish_auth 0 2024-04-19T01:42:27.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06967fc640 10 monclient: _check_auth_tickets 2024-04-19T01:42:27.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:27.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06967fc640 1 -- 172.21.15.62:0/1709664659 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f069c00cb50 con 0x7f06a00a60b0 2024-04-19T01:42:27.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06a8205640 5 monclient: authenticate success, global_id 4292 2024-04-19T01:42:27.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06a8205640 10 monclient: _renew_subs 2024-04-19T01:42:27.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06a8205640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:27.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06a8205640 1 -- 172.21.15.62:0/1709664659 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06a013de40 con 0x7f06a00a60b0 2024-04-19T01:42:27.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06967fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:27.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06967fc640 1 -- 172.21.15.62:0/1709664659 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069c013470 con 0x7f06a00a60b0 2024-04-19T01:42:27.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:27.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:27.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06967fc640 10 monclient: dump: 2024-04-19T01:42:27.329 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:27.329 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.330 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.330 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.330 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:27.330 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.330 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:27.330 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:27.330 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:27.330 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f0697fff640 4 set_mon_vals no callback set 2024-04-19T01:42:27.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06a8205640 10 monclient: _renew_subs 2024-04-19T01:42:27.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06a8205640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:27.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.314+0000 7f06a8205640 1 -- 172.21.15.62:0/1709664659 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f06a0120220 con 0x7f06a00a60b0 2024-04-19T01:42:27.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.318+0000 7f06967fc640 1 -- 172.21.15.62:0/1709664659 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f069c01a070 con 0x7f06a00a60b0 2024-04-19T01:42:27.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.318+0000 7f06967fc640 1 --2- 172.21.15.62:0/1709664659 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0684036470 0x7f0684038930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.318+0000 7f06967fc640 1 -- 172.21.15.62:0/1709664659 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4346+0+0 (secure 0 0 0) 0x7f069c0493c0 con 0x7f06a00a60b0 2024-04-19T01:42:27.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.318+0000 7f06a8205640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:27.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.318+0000 7f06a8205640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:27.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.318+0000 7f06a8205640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:27.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.318+0000 7f06a8205640 1 -- 172.21.15.62:0/1709664659 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06a0068ab0 con 0x7f06a00a60b0 2024-04-19T01:42:27.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.318+0000 7f06a57a0640 1 --2- 172.21.15.62:0/1709664659 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0684036470 0x7f0684038930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.322+0000 7f06a57a0640 10 monclient: get_auth_request con 0x7f0684036470 auth_method 0 2024-04-19T01:42:27.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.322+0000 7f06967fc640 1 -- 172.21.15.62:0/1709664659 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f069c013e10 con 0x7f06a00a60b0 2024-04-19T01:42:27.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.322+0000 7f06967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:27.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.322+0000 7f06967fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:27.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.322+0000 7f06a57a0640 1 --2- 172.21.15.62:0/1709664659 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0684036470 0x7f0684038930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f06a013dae0 tx=0x7f0690007e40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:27.338 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.5 2024-04-19T01:42:27.338 DEBUG:teuthology.parallel:result is None 2024-04-19T01:42:27.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.358+0000 7f445b11d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-19T01:42:27.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.358+0000 7f445b11d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-19T01:42:27.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.358+0000 7f445b11d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:27.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.358+0000 7f445b11d640 1 -- 172.21.15.62:0/1134699292 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f4454068710 con 0x7f44540a60b0 2024-04-19T01:42:27.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.358+0000 7f44517fa640 1 -- 172.21.15.62:0/1134699292 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f444800fd60 con 0x7f44540a60b0 2024-04-19T01:42:27.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.358+0000 7f44517fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-19T01:42:27.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.358+0000 7f44517fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:27.358 INFO:teuthology.orchestra.run.smithi062.stdout:47244640260 2024-04-19T01:42:27.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.358+0000 7f445b11d640 1 -- 172.21.15.62:0/1134699292 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4438036470 msgr2=0x7f4438038930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.358+0000 7f445b11d640 1 --2- 172.21.15.62:0/1134699292 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4438036470 0x7f4438038930 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f44540a1380 tx=0x7f4440007e40 comp rx=0 tx=0).stop 2024-04-19T01:42:27.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.358+0000 7f445b11d640 10 monclient: shutdown 2024-04-19T01:42:27.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.358+0000 7f445b11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:27.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.358+0000 7f445b11d640 1 -- 172.21.15.62:0/1134699292 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f44540a60b0 msgr2=0x7f445413ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.358+0000 7f445b11d640 1 --2- 172.21.15.62:0/1134699292 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f44540a60b0 0x7f445413ef00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f444800b550 tx=0x7f444800ba10 comp rx=0 tx=0).stop 2024-04-19T01:42:27.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.358+0000 7f445b11d640 1 -- 172.21.15.62:0/1134699292 shutdown_connections 2024-04-19T01:42:27.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.358+0000 7f445b11d640 1 --2- 172.21.15.62:0/1134699292 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44541380a0 0x7f44541389f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.358+0000 7f445b11d640 1 --2- 172.21.15.62:0/1134699292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4454137780 0x7f44541384b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.358+0000 7f445b11d640 1 --2- 172.21.15.62:0/1134699292 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4438036470 0x7f4438038930 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.358+0000 7f445b11d640 1 --2- 172.21.15.62:0/1134699292 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f44540a60b0 0x7f445413ef00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.358+0000 7f445b11d640 1 -- 172.21.15.62:0/1134699292 >> 172.21.15.62:0/1134699292 conn(0x7f44540aa1d0 msgr2=0x7f44540a79d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:27.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.358+0000 7f445b11d640 1 -- 172.21.15.62:0/1134699292 shutdown_connections 2024-04-19T01:42:27.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.358+0000 7f445b11d640 1 -- 172.21.15.62:0/1134699292 wait complete. 2024-04-19T01:42:27.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.378+0000 7facd6f6b640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-19T01:42:27.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.378+0000 7facd6f6b640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-19T01:42:27.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.378+0000 7facd6f6b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:27.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.378+0000 7facd6f6b640 1 -- 172.21.15.62:0/1047736057 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7facd0141730 con 0x7facd00726e0 2024-04-19T01:42:27.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.378+0000 7faccd7fa640 1 -- 172.21.15.62:0/1047736057 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7facc401a760 con 0x7facd00726e0 2024-04-19T01:42:27.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.378+0000 7faccd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-19T01:42:27.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.378+0000 7faccd7fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:27.379 INFO:teuthology.orchestra.run.smithi062.stdout:51539607554 2024-04-19T01:42:27.382 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.0 2024-04-19T01:42:27.382 DEBUG:teuthology.parallel:result is None 2024-04-19T01:42:27.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.382+0000 7facd6f6b640 1 -- 172.21.15.62:0/1047736057 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7facb4036420 msgr2=0x7facb40388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.382+0000 7facd6f6b640 1 --2- 172.21.15.62:0/1047736057 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7facb4036420 0x7facb40388e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7facd0128a30 tx=0x7facbc003700 comp rx=0 tx=0).stop 2024-04-19T01:42:27.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.382+0000 7facd6f6b640 10 monclient: shutdown 2024-04-19T01:42:27.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.382+0000 7facd6f6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:27.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.382+0000 7facd6f6b640 1 -- 172.21.15.62:0/1047736057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7facd00726e0 msgr2=0x7facd01d35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.382+0000 7facd6f6b640 1 --2- 172.21.15.62:0/1047736057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7facd00726e0 0x7facd01d35c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7facc400c3e0 tx=0x7facc400c8a0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.382+0000 7facd6f6b640 1 -- 172.21.15.62:0/1047736057 shutdown_connections 2024-04-19T01:42:27.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.382+0000 7facd6f6b640 1 --2- 172.21.15.62:0/1047736057 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7facb4036420 0x7facb40388e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.382+0000 7facd6f6b640 1 --2- 172.21.15.62:0/1047736057 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7facd0072b00 0x7facd0143c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.382+0000 7facd6f6b640 1 --2- 172.21.15.62:0/1047736057 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7facd01355e0 0x7facd01268c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.382+0000 7facd6f6b640 1 --2- 172.21.15.62:0/1047736057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7facd00726e0 0x7facd01d35c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.382+0000 7facd6f6b640 1 -- 172.21.15.62:0/1047736057 >> 172.21.15.62:0/1047736057 conn(0x7facd00a8030 msgr2=0x7facd00a2830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:27.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.382+0000 7facd6f6b640 1 -- 172.21.15.62:0/1047736057 shutdown_connections 2024-04-19T01:42:27.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.382+0000 7facd6f6b640 1 -- 172.21.15.62:0/1047736057 wait complete. 2024-04-19T01:42:27.403 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607554 got 51539607554 for osd.4 2024-04-19T01:42:27.403 DEBUG:teuthology.parallel:result is None 2024-04-19T01:42:27.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.414+0000 7fb6d37fe640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-19T01:42:27.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.414+0000 7fb6d37fe640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-19T01:42:27.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.414+0000 7fb6d37fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:27.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.414+0000 7fb6d37fe640 1 -- 172.21.15.62:0/633346152 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fb6c0005380 con 0x7fb6e813cc60 2024-04-19T01:42:27.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.414+0000 7fb6ddffb640 1 -- 172.21.15.62:0/633346152 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb6e809c2e0 con 0x7fb6e813cc60 2024-04-19T01:42:27.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.414+0000 7fb6ddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-19T01:42:27.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.414+0000 7fb6ddffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:27.415 INFO:teuthology.orchestra.run.smithi062.stdout:47244640260 2024-04-19T01:42:27.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.414+0000 7fb6d37fe640 1 -- 172.21.15.62:0/633346152 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fb6cc036420 msgr2=0x7fb6cc0388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.414+0000 7fb6d37fe640 1 --2- 172.21.15.62:0/633346152 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fb6cc036420 0x7fb6cc0388e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb6e4007fd0 tx=0x7fb6e4009040 comp rx=0 tx=0).stop 2024-04-19T01:42:27.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.414+0000 7fb6d37fe640 10 monclient: shutdown 2024-04-19T01:42:27.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.414+0000 7fb6d37fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:27.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.414+0000 7fb6d37fe640 1 -- 172.21.15.62:0/633346152 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb6e813cc60 msgr2=0x7fb6e81364c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.414+0000 7fb6d37fe640 1 --2- 172.21.15.62:0/633346152 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb6e813cc60 0x7fb6e81364c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb6d80097c0 tx=0x7fb6d8018fb0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.414+0000 7fb6d37fe640 1 -- 172.21.15.62:0/633346152 shutdown_connections 2024-04-19T01:42:27.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.414+0000 7fb6d37fe640 1 --2- 172.21.15.62:0/633346152 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fb6cc036420 0x7fb6cc0388e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.414+0000 7fb6d37fe640 1 --2- 172.21.15.62:0/633346152 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb6e813cc60 0x7fb6e81364c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.414+0000 7fb6d37fe640 1 --2- 172.21.15.62:0/633346152 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6e809e9d0 0x7fb6e809ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.414+0000 7fb6d37fe640 1 --2- 172.21.15.62:0/633346152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6e8137780 0x7fb6e81cf630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.414+0000 7fb6d37fe640 1 -- 172.21.15.62:0/633346152 >> 172.21.15.62:0/633346152 conn(0x7fb6e80aa1d0 msgr2=0x7fb6e80a79d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:27.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.414+0000 7fb6d37fe640 1 -- 172.21.15.62:0/633346152 shutdown_connections 2024-04-19T01:42:27.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.414+0000 7fb6d37fe640 1 -- 172.21.15.62:0/633346152 wait complete. 2024-04-19T01:42:27.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.430+0000 7f505ffff640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-19T01:42:27.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.430+0000 7f505ffff640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-19T01:42:27.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.430+0000 7f505ffff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:27.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.430+0000 7f505ffff640 1 -- 172.21.15.62:0/231798331 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f505802d810 con 0x7f50580c10e0 2024-04-19T01:42:27.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.430+0000 7f50477fe640 1 -- 172.21.15.62:0/231798331 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5058023ac0 con 0x7f50580c10e0 2024-04-19T01:42:27.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.430+0000 7f50477fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-19T01:42:27.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.430+0000 7f50477fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:27.432 INFO:teuthology.orchestra.run.smithi062.stdout:47244640260 2024-04-19T01:42:27.432 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640260 for osd.3 2024-04-19T01:42:27.433 DEBUG:teuthology.parallel:result is None 2024-04-19T01:42:27.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.430+0000 7f505ffff640 1 -- 172.21.15.62:0/231798331 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f504c036470 msgr2=0x7f504c038930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.430+0000 7f505ffff640 1 --2- 172.21.15.62:0/231798331 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f504c036470 0x7f504c038930 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f50500050b0 tx=0x7f5050003700 comp rx=0 tx=0).stop 2024-04-19T01:42:27.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.430+0000 7f505ffff640 10 monclient: shutdown 2024-04-19T01:42:27.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.430+0000 7f505ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:27.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.430+0000 7f505ffff640 1 -- 172.21.15.62:0/231798331 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f50580c10e0 msgr2=0x7f50580c0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.430+0000 7f505ffff640 1 --2- 172.21.15.62:0/231798331 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f50580c10e0 0x7f50580c0620 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5060069920 tx=0x7f5060063510 comp rx=0 tx=0).stop 2024-04-19T01:42:27.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.430+0000 7f505ffff640 1 -- 172.21.15.62:0/231798331 shutdown_connections 2024-04-19T01:42:27.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.430+0000 7f505ffff640 1 --2- 172.21.15.62:0/231798331 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f504c036470 0x7f504c038930 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.430+0000 7f505ffff640 1 --2- 172.21.15.62:0/231798331 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f50580c10e0 0x7f50580c0620 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.430+0000 7f505ffff640 1 --2- 172.21.15.62:0/231798331 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5058027b60 0x7f50580c00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.430+0000 7f505ffff640 1 --2- 172.21.15.62:0/231798331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5058027780 0x7f5058153b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.430+0000 7f505ffff640 1 -- 172.21.15.62:0/231798331 >> 172.21.15.62:0/231798331 conn(0x7f5058006800 msgr2=0x7f505802c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:27.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.430+0000 7f505ffff640 1 -- 172.21.15.62:0/231798331 shutdown_connections 2024-04-19T01:42:27.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.430+0000 7f505ffff640 1 -- 172.21.15.62:0/231798331 wait complete. 2024-04-19T01:42:27.448 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.2 2024-04-19T01:42:27.448 DEBUG:teuthology.parallel:result is None 2024-04-19T01:42:27.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.518+0000 7f06a8205640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-19T01:42:27.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.518+0000 7f06a8205640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-19T01:42:27.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.522+0000 7f06a8205640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:27.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.522+0000 7f06a8205640 1 -- 172.21.15.62:0/1709664659 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f06a0068710 con 0x7f06a00a60b0 2024-04-19T01:42:27.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.522+0000 7f06967fc640 1 -- 172.21.15.62:0/1709664659 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f069c0490c0 con 0x7f06a00a60b0 2024-04-19T01:42:27.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.522+0000 7f06967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-19T01:42:27.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.522+0000 7f06967fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:27.523 INFO:teuthology.orchestra.run.smithi062.stdout:47244640260 2024-04-19T01:42:27.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.522+0000 7f06a8205640 1 -- 172.21.15.62:0/1709664659 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0684036470 msgr2=0x7f0684038930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.522+0000 7f06a8205640 1 --2- 172.21.15.62:0/1709664659 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0684036470 0x7f0684038930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f06a013dae0 tx=0x7f0690007e40 comp rx=0 tx=0).stop 2024-04-19T01:42:27.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.522+0000 7f06a8205640 10 monclient: shutdown 2024-04-19T01:42:27.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.522+0000 7f06a8205640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:27.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.522+0000 7f06a8205640 1 -- 172.21.15.62:0/1709664659 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06a00a60b0 msgr2=0x7f06a01cf5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.522+0000 7f06a8205640 1 --2- 172.21.15.62:0/1709664659 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06a00a60b0 0x7f06a01cf5a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f069c00e8b0 tx=0x7f069c00ed70 comp rx=0 tx=0).stop 2024-04-19T01:42:27.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.522+0000 7f06a8205640 1 -- 172.21.15.62:0/1709664659 shutdown_connections 2024-04-19T01:42:27.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.522+0000 7f06a8205640 1 --2- 172.21.15.62:0/1709664659 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06a01380a0 0x7f06a013c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.522+0000 7f06a8205640 1 --2- 172.21.15.62:0/1709664659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f06a0137780 0x7f06a01cfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.522+0000 7f06a8205640 1 --2- 172.21.15.62:0/1709664659 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0684036470 0x7f0684038930 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.522+0000 7f06a8205640 1 --2- 172.21.15.62:0/1709664659 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f06a00a60b0 0x7f06a01cf5a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.522+0000 7f06a8205640 1 -- 172.21.15.62:0/1709664659 >> 172.21.15.62:0/1709664659 conn(0x7f06a00aa1d0 msgr2=0x7f06a00a79d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:27.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.522+0000 7f06a8205640 1 -- 172.21.15.62:0/1709664659 shutdown_connections 2024-04-19T01:42:27.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.522+0000 7f06a8205640 1 -- 172.21.15.62:0/1709664659 wait complete. 2024-04-19T01:42:27.539 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.1 2024-04-19T01:42:27.539 DEBUG:teuthology.parallel:result is None 2024-04-19T01:42:27.539 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-19T01:42:27.539 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:42:27.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.654+0000 7f2dcac9a640 10 monclient: get_monmap_and_config 2024-04-19T01:42:27.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.654+0000 7f2dcac9a640 10 monclient: build_initial_monmap 2024-04-19T01:42:27.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.654+0000 7f2dcac9a640 10 monclient: monmap: 2024-04-19T01:42:27.656 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:27.656 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.656 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:27.660568+0000 2024-04-19T01:42:27.656 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:27.660568+0000 2024-04-19T01:42:27.656 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:27.656 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.657 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:27.657 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:27.657 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:27.657 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:27.657 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:27.657 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.654+0000 7f2dcac9a640 1 Processor -- start 2024-04-19T01:42:27.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.654+0000 7f2dcac9a640 1 -- start start 2024-04-19T01:42:27.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.654+0000 7f2dcac9a640 10 monclient: init 2024-04-19T01:42:27.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.654+0000 7f2dcac9a640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:27.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.654+0000 7f2dcac9a640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-04-19T01:42:27.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.654+0000 7f2dcac9a640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2dc40a0b60 0x7f2dc40990c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.654+0000 7f2dcac9a640 10 monclient(hunting): picked mon.noname-a con 0x7f2dc40a0b60 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:27.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.654+0000 7f2dcac9a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2dc4099600 0x7f2dc4099a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.654+0000 7f2dcac9a640 10 monclient(hunting): picked mon.noname-b con 0x7f2dc4099600 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:27.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.654+0000 7f2dcac9a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2dc40a0f40 addr v1:172.21.15.71:6789/0 2024-04-19T01:42:27.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dcac9a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dc409d0c0 con 0x7f2dc40a0f40 2024-04-19T01:42:27.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dcac9a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dcac9a640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f2dc4099f50 con 0x7f2dc40a0b60 2024-04-19T01:42:27.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dcac9a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dcac9a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f2dc409d790 con 0x7f2dc4099600 2024-04-19T01:42:27.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dcac9a640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:27.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dcac9a640 10 monclient(hunting): authenticate will time out at 933.480033s 2024-04-19T01:42:27.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dc3fff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2dc4099600 0x7f2dc4099a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dc8a36640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2dc40a0b60 0x7f2dc40990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dc8a36640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2dc40a0b60 0x7f2dc40990c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36404/0 (socket says 172.21.15.62:36404) 2024-04-19T01:42:27.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dc8a36640 1 -- 172.21.15.62:0/1568117872 learned_addr learned my addr 172.21.15.62:0/1568117872 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:27.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dc8a36640 10 monclient(hunting): get_auth_request con 0x7f2dc40a0b60 auth_method 0 2024-04-19T01:42:27.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dc3fff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2dc4099600 0x7f2dc4099a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60476/0 (socket says 172.21.15.62:60476) 2024-04-19T01:42:27.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dc8a36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:27.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dc8a36640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:27.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dc8a36640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:27.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dc3fff640 10 monclient(hunting): get_auth_request con 0x7f2dc4099600 auth_method 0 2024-04-19T01:42:27.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:27.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dc3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:27.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:27.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dc8a36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:27.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dc8a36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:27.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dc8a36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:27.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dc3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:27.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dc3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:27.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dc3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:27.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dc8a36640 10 monclient(hunting): handle_auth_done global_id 4293 payload 274 2024-04-19T01:42:27.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dc8a36640 1 -- 172.21.15.62:0/1568117872 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2dc4099600 msgr2=0x7f2dc4099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:27.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dc8a36640 1 --2- 172.21.15.62:0/1568117872 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2dc4099600 0x7f2dc4099a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dc8a36640 1 -- 172.21.15.62:0/1568117872 >> v1:172.21.15.71:6789/0 conn(0x7f2dc40a0f40 legacy=0x7f2dc40a1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:27.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dc8a36640 10 monclient: _finish_hunting 0 2024-04-19T01:42:27.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dc8a36640 1 monclient: found mon.noname-a 2024-04-19T01:42:27.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dc8a36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:27.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dc8a36640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:27.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dc8a36640 1 -- 172.21.15.62:0/1568117872 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dc409fbb0 con 0x7f2dc40a0b60 2024-04-19T01:42:27.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.658+0000 7f2dc3fff640 1 --2- 172.21.15.62:0/1568117872 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2dc4099600 0x7f2dc4099a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:42:27.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc8a36640 1 -- 172.21.15.62:0/1568117872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dc40a0b60 msgr2=0x7f2dc40990c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:27.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc8a36640 1 --2- 172.21.15.62:0/1568117872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dc40a0b60 0x7f2dc40990c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2db4009870 tx=0x7f2db401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db53d7034d46216 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:27.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc37fe640 1 -- 172.21.15.62:0/1568117872 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db40ab620 con 0x7f2dc40a0b60 2024-04-19T01:42:27.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:27.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:27.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc37fe640 10 monclient: dump: 2024-04-19T01:42:27.670 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:27.670 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.670 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.670 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.670 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:27.670 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.670 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:27.671 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:27.671 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:27.671 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc37fe640 10 monclient: _finish_auth 0 2024-04-19T01:42:27.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc37fe640 10 monclient: _check_auth_tickets 2024-04-19T01:42:27.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:27.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc37fe640 1 -- 172.21.15.62:0/1568117872 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2db40b0070 con 0x7f2dc40a0b60 2024-04-19T01:42:27.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 5 monclient: authenticate success, global_id 4293 2024-04-19T01:42:27.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:42:27.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc37fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:27.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc37fe640 1 -- 172.21.15.62:0/1568117872 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db40ab9e0 con 0x7f2dc40a0b60 2024-04-19T01:42:27.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:27.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 4 set_mon_vals no callback set 2024-04-19T01:42:27.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:27.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:27.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc37fe640 10 monclient: dump: 2024-04-19T01:42:27.672 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:27.672 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.672 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.672 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.672 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:27.672 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.672 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:27.672 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:27.672 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:27.672 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 10 monclient: shutdown 2024-04-19T01:42:27.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:27.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 1 -- 172.21.15.62:0/1568117872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dc40a0b60 msgr2=0x7f2dc40990c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 1 --2- 172.21.15.62:0/1568117872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dc40a0b60 0x7f2dc40990c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2db4009870 tx=0x7f2db401c480 comp rx=0 tx=0).stop 2024-04-19T01:42:27.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 1 -- 172.21.15.62:0/1568117872 shutdown_connections 2024-04-19T01:42:27.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 1 --2- 172.21.15.62:0/1568117872 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2dc4099600 0x7f2dc4099a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 1 --2- 172.21.15.62:0/1568117872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dc40a0b60 0x7f2dc40990c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 1 -- 172.21.15.62:0/1568117872 >> 172.21.15.62:0/1568117872 conn(0x7f2dc40a6160 msgr2=0x7f2dc40a6540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:27.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 1 -- 172.21.15.62:0/1568117872 shutdown_connections 2024-04-19T01:42:27.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 1 -- 172.21.15.62:0/1568117872 wait complete. 2024-04-19T01:42:27.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 10 monclient: build_initial_monmap 2024-04-19T01:42:27.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 10 monclient: monmap: 2024-04-19T01:42:27.673 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:27.673 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:27.673 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:27.673 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:27.673 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:27.673 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.673 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:27.673 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:27.673 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:27.673 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 1 Processor -- start 2024-04-19T01:42:27.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 1 -- start start 2024-04-19T01:42:27.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 10 monclient: init 2024-04-19T01:42:27.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:27.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-19T01:42:27.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2dc4099600 0x7f2dc41cf7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 10 monclient(hunting): picked mon.noname-b con 0x7f2dc4099600 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:27.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2dc40a0b60 0x7f2dc40983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 10 monclient(hunting): picked mon.noname-c con 0x7f2dc40a0b60 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:27.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dc40a0f40 0x7f2dc4098910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 10 monclient(hunting): picked mon.noname-a con 0x7f2dc40a0f40 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:27.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2dc41400d0 con 0x7f2dc40a0f40 2024-04-19T01:42:27.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f2dc413ff50 con 0x7f2dc4099600 2024-04-19T01:42:27.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f2dc4140250 con 0x7f2dc40a0b60 2024-04-19T01:42:27.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:27.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dcac9a640 10 monclient(hunting): authenticate will time out at 933.487368s 2024-04-19T01:42:27.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc9237640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dc40a0f40 0x7f2dc4098910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc3fff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2dc40a0b60 0x7f2dc40983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc9237640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dc40a0f40 0x7f2dc4098910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36406/0 (socket says 172.21.15.62:36406) 2024-04-19T01:42:27.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc9237640 1 -- 172.21.15.62:0/240307664 learned_addr learned my addr 172.21.15.62:0/240307664 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:27.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc9237640 10 monclient(hunting): get_auth_request con 0x7f2dc40a0f40 auth_method 0 2024-04-19T01:42:27.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc9237640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:27.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc9237640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:27.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc9237640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:27.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc3fff640 10 monclient(hunting): get_auth_request con 0x7f2dc40a0b60 auth_method 0 2024-04-19T01:42:27.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:27.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:27.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:27.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc9237640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:27.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc9237640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:27.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc9237640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:27.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:27.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:27.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:27.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc8a36640 1 --2- 172.21.15.62:0/240307664 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2dc4099600 0x7f2dc41cf7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc9237640 10 monclient(hunting): handle_auth_done global_id 4296 payload 824 2024-04-19T01:42:27.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc9237640 1 -- 172.21.15.62:0/240307664 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2dc40a0b60 msgr2=0x7f2dc40983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc9237640 1 --2- 172.21.15.62:0/240307664 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2dc40a0b60 0x7f2dc40983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc9237640 1 -- 172.21.15.62:0/240307664 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2dc4099600 msgr2=0x7f2dc41cf7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc9237640 1 --2- 172.21.15.62:0/240307664 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2dc4099600 0x7f2dc41cf7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc9237640 10 monclient: _finish_hunting 0 2024-04-19T01:42:27.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc9237640 1 monclient: found mon.noname-a 2024-04-19T01:42:27.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc9237640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:27.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc9237640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:27.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc9237640 1 -- 172.21.15.62:0/240307664 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dc409f880 con 0x7f2dc40a0f40 2024-04-19T01:42:27.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.662+0000 7f2dc3fff640 1 --2- 172.21.15.62:0/240307664 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2dc40a0b60 0x7f2dc40983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:42:27.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.666+0000 7f2dc9237640 1 -- 172.21.15.62:0/240307664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dc40a0f40 msgr2=0x7f2dc4098910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:27.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.666+0000 7f2dc9237640 1 --2- 172.21.15.62:0/240307664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dc40a0f40 0x7f2dc4098910 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc00c7b0 tx=0x7f2dbc00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:27.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.666+0000 7f2dc17fa640 1 -- 172.21.15.62:0/240307664 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc00ac60 con 0x7f2dc40a0f40 2024-04-19T01:42:27.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.666+0000 7f2dc17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:27.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.666+0000 7f2dc17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:27.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.666+0000 7f2dc17fa640 10 monclient: dump: 2024-04-19T01:42:27.681 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:27.681 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.681 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.681 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.682 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:27.682 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.682 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:27.682 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:27.682 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:27.682 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.666+0000 7f2dc17fa640 10 monclient: _finish_auth 0 2024-04-19T01:42:27.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.666+0000 7f2dc17fa640 10 monclient: _check_auth_tickets 2024-04-19T01:42:27.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.666+0000 7f2dc17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:27.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.666+0000 7f2dc17fa640 1 -- 172.21.15.62:0/240307664 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dbc00ae00 con 0x7f2dc40a0f40 2024-04-19T01:42:27.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.666+0000 7f2dc17fa640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:27.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.666+0000 7f2dc17fa640 1 -- 172.21.15.62:0/240307664 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc004bb0 con 0x7f2dc40a0f40 2024-04-19T01:42:27.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.666+0000 7f2dc17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:27.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.666+0000 7f2dc17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:27.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.666+0000 7f2dc17fa640 10 monclient: dump: 2024-04-19T01:42:27.684 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:27.684 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.684 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.684 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.685 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:27.685 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.685 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:27.685 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:27.685 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:27.685 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.666+0000 7f2dcac9a640 5 monclient: authenticate success, global_id 4296 2024-04-19T01:42:27.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.666+0000 7f2dcac9a640 10 monclient: _renew_subs 2024-04-19T01:42:27.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.666+0000 7f2dcac9a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:27.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.666+0000 7f2dcac9a640 1 -- 172.21.15.62:0/240307664 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dc40a5080 con 0x7f2dc40a0f40 2024-04-19T01:42:27.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.666+0000 7f2dc2ffd640 4 set_mon_vals no callback set 2024-04-19T01:42:27.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.666+0000 7f2dcac9a640 10 monclient: _renew_subs 2024-04-19T01:42:27.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.666+0000 7f2dcac9a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:27.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.666+0000 7f2dcac9a640 1 -- 172.21.15.62:0/240307664 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dc40933a0 con 0x7f2dc40a0f40 2024-04-19T01:42:27.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.666+0000 7f2dcac9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:27.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.666+0000 7f2dcac9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:27.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.666+0000 7f2dcac9a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:27.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.666+0000 7f2dc17fa640 1 -- 172.21.15.62:0/240307664 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f2dbc005030 con 0x7f2dc40a0f40 2024-04-19T01:42:27.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.666+0000 7f2dcac9a640 1 -- 172.21.15.62:0/240307664 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dc4068ab0 con 0x7f2dc40a0f40 2024-04-19T01:42:27.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.666+0000 7f2dc17fa640 1 --2- 172.21.15.62:0/240307664 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f2db0036470 0x7f2db0038930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.666+0000 7f2dc17fa640 1 -- 172.21.15.62:0/240307664 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4346+0+0 (secure 0 0 0) 0x7f2dbc049590 con 0x7f2dc40a0f40 2024-04-19T01:42:27.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.666+0000 7f2dc8a36640 1 --2- 172.21.15.62:0/240307664 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f2db0036470 0x7f2db0038930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.666+0000 7f2dc8a36640 10 monclient: get_auth_request con 0x7f2db0036470 auth_method 0 2024-04-19T01:42:27.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.670+0000 7f2dc8a36640 1 --2- 172.21.15.62:0/240307664 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f2db0036470 0x7f2db0038930 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2dc406bec0 tx=0x7f2db4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:27.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.670+0000 7f2dc17fa640 1 -- 172.21.15.62:0/240307664 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2dbc020070 con 0x7f2dc40a0f40 2024-04-19T01:42:27.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.670+0000 7f2dc17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:27.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.670+0000 7f2dc17fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:27.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.822+0000 7f2dcac9a640 1 -- 172.21.15.62:0/240307664 --> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2dc4092460 con 0x7f2db0036470 2024-04-19T01:42:27.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.826+0000 7f2dc17fa640 1 -- 172.21.15.62:0/240307664 <== mgr.4108 v2:172.21.15.71:6824/2133533992 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33750 (secure 0 0 0) 0x7f2dc4092460 con 0x7f2db0036470 2024-04-19T01:42:27.829 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:42:27.830 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-04-19T01:42:27.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.830+0000 7f2dcac9a640 1 -- 172.21.15.62:0/240307664 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f2db0036470 msgr2=0x7f2db0038930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.830+0000 7f2dcac9a640 1 --2- 172.21.15.62:0/240307664 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f2db0036470 0x7f2db0038930 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2dc406bec0 tx=0x7f2db4008040 comp rx=0 tx=0).stop 2024-04-19T01:42:27.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.830+0000 7f2dcac9a640 10 monclient: shutdown 2024-04-19T01:42:27.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.830+0000 7f2dcac9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:27.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.830+0000 7f2dcac9a640 1 -- 172.21.15.62:0/240307664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dc40a0f40 msgr2=0x7f2dc4098910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.830+0000 7f2dcac9a640 1 --2- 172.21.15.62:0/240307664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dc40a0f40 0x7f2dc4098910 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc00c7b0 tx=0x7f2dbc00cc70 comp rx=0 tx=0).stop 2024-04-19T01:42:27.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.830+0000 7f2dcac9a640 1 -- 172.21.15.62:0/240307664 shutdown_connections 2024-04-19T01:42:27.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.830+0000 7f2dcac9a640 1 --2- 172.21.15.62:0/240307664 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f2db0036470 0x7f2db0038930 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.830+0000 7f2dcac9a640 1 --2- 172.21.15.62:0/240307664 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2dc40a0b60 0x7f2dc40983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.830+0000 7f2dcac9a640 1 --2- 172.21.15.62:0/240307664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dc40a0f40 0x7f2dc4098910 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.830+0000 7f2dcac9a640 1 --2- 172.21.15.62:0/240307664 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2dc4099600 0x7f2dc41cf7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.830+0000 7f2dcac9a640 1 -- 172.21.15.62:0/240307664 >> 172.21.15.62:0/240307664 conn(0x7f2dc40a6160 msgr2=0x7f2dc409df60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:27.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.830+0000 7f2dcac9a640 1 -- 172.21.15.62:0/240307664 shutdown_connections 2024-04-19T01:42:27.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.830+0000 7f2dcac9a640 1 -- 172.21.15.62:0/240307664 wait complete. 2024-04-19T01:42:27.846 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-04-19T01:42:26.704787+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":101,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164112,"kb_used_data":1856,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066928,"statfs":{"total":579820584960,"available":579652534272,"internally_reserved":0,"allocated":1900544,"data_stored":1317428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.681345"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.163776+0000","last_change":"2024-04-19T01:42:24.162228+0000","last_active":"2024-04-19T01:42:24.163776+0000","last_peered":"2024-04-19T01:42:24.163776+0000","last_clean":"2024-04-19T01:42:24.163776+0000","last_became_active":"2024-04-19T01:42:24.161645+0000","last_became_peered":"2024-04-19T01:42:24.161645+0000","last_unstale":"2024-04-19T01:42:24.163776+0000","last_undegraded":"2024-04-19T01:42:24.163776+0000","last_fullsized":"2024-04-19T01:42:24.163776+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:08:44.489901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.141485+0000","last_change":"2024-04-19T01:42:24.132804+0000","last_active":"2024-04-19T01:42:24.141485+0000","last_peered":"2024-04-19T01:42:24.141485+0000","last_clean":"2024-04-19T01:42:24.141485+0000","last_became_active":"2024-04-19T01:42:24.131937+0000","last_became_peered":"2024-04-19T01:42:24.131937+0000","last_unstale":"2024-04-19T01:42:24.141485+0000","last_undegraded":"2024-04-19T01:42:24.141485+0000","last_fullsized":"2024-04-19T01:42:24.141485+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T06:51:05.320754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.139253+0000","last_change":"2024-04-19T01:42:24.135590+0000","last_active":"2024-04-19T01:42:24.139253+0000","last_peered":"2024-04-19T01:42:24.139253+0000","last_clean":"2024-04-19T01:42:24.139253+0000","last_became_active":"2024-04-19T01:42:24.134918+0000","last_became_peered":"2024-04-19T01:42:24.134918+0000","last_unstale":"2024-04-19T01:42:24.139253+0000","last_undegraded":"2024-04-19T01:42:24.139253+0000","last_fullsized":"2024-04-19T01:42:24.139253+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T08:09:26.534746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.141458+0000","last_change":"2024-04-19T01:42:24.132575+0000","last_active":"2024-04-19T01:42:24.141458+0000","last_peered":"2024-04-19T01:42:24.141458+0000","last_clean":"2024-04-19T01:42:24.141458+0000","last_became_active":"2024-04-19T01:42:24.131894+0000","last_became_peered":"2024-04-19T01:42:24.131894+0000","last_unstale":"2024-04-19T01:42:24.141458+0000","last_undegraded":"2024-04-19T01:42:24.141458+0000","last_fullsized":"2024-04-19T01:42:24.141458+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:25:51.912215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.148595+0000","last_change":"2024-04-19T01:42:24.145699+0000","last_active":"2024-04-19T01:42:24.148595+0000","last_peered":"2024-04-19T01:42:24.148595+0000","last_clean":"2024-04-19T01:42:24.148595+0000","last_became_active":"2024-04-19T01:42:24.144746+0000","last_became_peered":"2024-04-19T01:42:24.144746+0000","last_unstale":"2024-04-19T01:42:24.148595+0000","last_undegraded":"2024-04-19T01:42:24.148595+0000","last_fullsized":"2024-04-19T01:42:24.148595+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T04:20:46.381920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.145097+0000","last_change":"2024-04-19T01:42:24.142754+0000","last_active":"2024-04-19T01:42:24.145097+0000","last_peered":"2024-04-19T01:42:24.145097+0000","last_clean":"2024-04-19T01:42:24.145097+0000","last_became_active":"2024-04-19T01:42:24.142309+0000","last_became_peered":"2024-04-19T01:42:24.142309+0000","last_unstale":"2024-04-19T01:42:24.145097+0000","last_undegraded":"2024-04-19T01:42:24.145097+0000","last_fullsized":"2024-04-19T01:42:24.145097+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:18:58.995723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.136907+0000","last_change":"2024-04-19T01:42:24.134484+0000","last_active":"2024-04-19T01:42:24.136907+0000","last_peered":"2024-04-19T01:42:24.136907+0000","last_clean":"2024-04-19T01:42:24.136907+0000","last_became_active":"2024-04-19T01:42:24.133282+0000","last_became_peered":"2024-04-19T01:42:24.133282+0000","last_unstale":"2024-04-19T01:42:24.136907+0000","last_undegraded":"2024-04-19T01:42:24.136907+0000","last_fullsized":"2024-04-19T01:42:24.136907+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T06:02:03.349002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.149236+0000","last_change":"2024-04-19T01:42:24.146550+0000","last_active":"2024-04-19T01:42:24.149236+0000","last_peered":"2024-04-19T01:42:24.149236+0000","last_clean":"2024-04-19T01:42:24.149236+0000","last_became_active":"2024-04-19T01:42:24.145667+0000","last_became_peered":"2024-04-19T01:42:24.145667+0000","last_unstale":"2024-04-19T01:42:24.149236+0000","last_undegraded":"2024-04-19T01:42:24.149236+0000","last_fullsized":"2024-04-19T01:42:24.149236+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:48:51.752906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":165,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.130442+0000","last_change":"2024-04-19T01:42:22.132251+0000","last_active":"2024-04-19T01:42:24.130442+0000","last_peered":"2024-04-19T01:42:24.130442+0000","last_clean":"2024-04-19T01:42:24.130442+0000","last_became_active":"2024-04-19T01:42:22.131432+0000","last_became_peered":"2024-04-19T01:42:22.131432+0000","last_unstale":"2024-04-19T01:42:24.130442+0000","last_undegraded":"2024-04-19T01:42:24.130442+0000","last_fullsized":"2024-04-19T01:42:24.130442+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:21.102542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:21.102542+0000","last_clean_scrub_stamp":"2024-04-19T01:42:21.102542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T06:47:12.046147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":101,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":101,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607554,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":139264,"data_stored":43262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":139264,"data_stored":43262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344164,"statfs":{"total":96636764160,"available":96608423936,"internally_reserved":0,"allocated":671744,"data_stored":572190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":139264,"data_stored":43262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":139264,"data_stored":43262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":671744,"data_stored":572190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:42:27.847 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:42:27.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.954+0000 7fe488483640 10 monclient: get_monmap_and_config 2024-04-19T01:42:27.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.954+0000 7fe488483640 10 monclient: build_initial_monmap 2024-04-19T01:42:27.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.954+0000 7fe488483640 10 monclient: monmap: 2024-04-19T01:42:27.956 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:27.956 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.956 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:27.960288+0000 2024-04-19T01:42:27.956 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:27.960288+0000 2024-04-19T01:42:27.956 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:27.956 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.956 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:27.956 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:27.956 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:27.957 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:27.957 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:27.957 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.954+0000 7fe488483640 1 Processor -- start 2024-04-19T01:42:27.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.954+0000 7fe488483640 1 -- start start 2024-04-19T01:42:27.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.954+0000 7fe488483640 10 monclient: init 2024-04-19T01:42:27.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.954+0000 7fe488483640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:27.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.954+0000 7fe488483640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-04-19T01:42:27.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.954+0000 7fe488483640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe4800a3410 0x7fe4800a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.954+0000 7fe488483640 10 monclient(hunting): picked mon.noname-a con 0x7fe4800a3410 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:27.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.954+0000 7fe488483640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe4800a2900 0x7fe4800a46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.954+0000 7fe488483640 10 monclient(hunting): picked mon.noname-c con 0x7fe4800a2900 addr v2:172.21.15.71:3300/0 2024-04-19T01:42:27.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.954+0000 7fe488483640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4800a2d50 0x7fe4800a4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.954+0000 7fe488483640 10 monclient(hunting): picked mon.noname-b con 0x7fe4800a2d50 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:27.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.954+0000 7fe488483640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.954+0000 7fe488483640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fe4800a1430 con 0x7fe4800a3410 2024-04-19T01:42:27.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.954+0000 7fe488483640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.954+0000 7fe488483640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fe4800a1710 con 0x7fe4800a2900 2024-04-19T01:42:27.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.954+0000 7fe488483640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.954+0000 7fe488483640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fe4800a19f0 con 0x7fe4800a2d50 2024-04-19T01:42:27.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.954+0000 7fe488483640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:27.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.954+0000 7fe488483640 10 monclient(hunting): authenticate will time out at 933.779740s 2024-04-19T01:42:27.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.954+0000 7fe486a20640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4800a2d50 0x7fe4800a4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.954+0000 7fe48621f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe4800a3410 0x7fe4800a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.954+0000 7fe486a20640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4800a2d50 0x7fe4800a4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60496/0 (socket says 172.21.15.62:60496) 2024-04-19T01:42:27.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.954+0000 7fe486a20640 1 -- 172.21.15.62:0/3846399123 learned_addr learned my addr 172.21.15.62:0/3846399123 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:27.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.954+0000 7fe48621f640 1 --2- 172.21.15.62:0/3846399123 >> v2:172.21.15.62:3300/0 conn(0x7fe4800a3410 0x7fe4800a37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36410/0 (socket says 172.21.15.62:36410) 2024-04-19T01:42:27.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.954+0000 7fe486a20640 10 monclient(hunting): get_auth_request con 0x7fe4800a2d50 auth_method 0 2024-04-19T01:42:27.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.954+0000 7fe486a20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:27.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.954+0000 7fe486a20640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:27.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.954+0000 7fe486a20640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:27.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.958+0000 7fe48621f640 10 monclient(hunting): get_auth_request con 0x7fe4800a3410 auth_method 0 2024-04-19T01:42:27.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.958+0000 7fe48621f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:27.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.958+0000 7fe48621f640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:27.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.958+0000 7fe48621f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:27.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.958+0000 7fe486a20640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:27.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.958+0000 7fe486a20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:27.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.958+0000 7fe486a20640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:27.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.958+0000 7fe48621f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:27.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.958+0000 7fe48621f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:27.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.958+0000 7fe48621f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:27.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.958+0000 7fe485a1e640 1 --2- 172.21.15.62:0/3846399123 >> v2:172.21.15.71:3300/0 conn(0x7fe4800a2900 0x7fe4800a46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.958+0000 7fe485a1e640 10 monclient(hunting): get_auth_request con 0x7fe4800a2900 auth_method 0 2024-04-19T01:42:27.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.958+0000 7fe485a1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:27.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.958+0000 7fe485a1e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:27.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.958+0000 7fe485a1e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:27.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.958+0000 7fe48621f640 10 monclient(hunting): handle_auth_done global_id 4299 payload 274 2024-04-19T01:42:27.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.958+0000 7fe48621f640 1 -- 172.21.15.62:0/3846399123 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4800a2d50 msgr2=0x7fe4800a4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:27.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.958+0000 7fe48621f640 1 --2- 172.21.15.62:0/3846399123 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4800a2d50 0x7fe4800a4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.958+0000 7fe48621f640 1 -- 172.21.15.62:0/3846399123 >> v2:172.21.15.71:3300/0 conn(0x7fe4800a2900 msgr2=0x7fe4800a46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:27.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.958+0000 7fe48621f640 1 --2- 172.21.15.62:0/3846399123 >> v2:172.21.15.71:3300/0 conn(0x7fe4800a2900 0x7fe4800a46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.958+0000 7fe48621f640 10 monclient: _finish_hunting 0 2024-04-19T01:42:27.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.958+0000 7fe48621f640 1 monclient: found mon.noname-a 2024-04-19T01:42:27.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.958+0000 7fe48621f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:27.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.958+0000 7fe48621f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:27.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.958+0000 7fe48621f640 1 -- 172.21.15.62:0/3846399123 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4800a3d30 con 0x7fe4800a3410 2024-04-19T01:42:27.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.958+0000 7fe486a20640 1 --2- 172.21.15.62:0/3846399123 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4800a2d50 0x7fe4800a4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:42:27.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe48621f640 1 -- 172.21.15.62:0/3846399123 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4800a3410 msgr2=0x7fe4800a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:27.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe48621f640 1 --2- 172.21.15.62:0/3846399123 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4800a3410 0x7fe4800a37f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe4800a10a0 tx=0x7fe47001c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bb36e357344f2af server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:27.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe48521d640 1 -- 172.21.15.62:0/3846399123 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4700b2070 con 0x7fe4800a3410 2024-04-19T01:42:27.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe48521d640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:27.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe48521d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:27.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe48521d640 10 monclient: dump: 2024-04-19T01:42:27.966 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:27.966 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.966 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.966 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.966 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:27.966 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.967 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:27.967 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:27.967 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:27.967 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe48521d640 10 monclient: _finish_auth 0 2024-04-19T01:42:27.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe48521d640 10 monclient: _check_auth_tickets 2024-04-19T01:42:27.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe48521d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:27.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe48521d640 1 -- 172.21.15.62:0/3846399123 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4700a5890 con 0x7fe4800a3410 2024-04-19T01:42:27.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 5 monclient: authenticate success, global_id 4299 2024-04-19T01:42:27.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:42:27.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe48521d640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:27.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:27.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe48521d640 1 -- 172.21.15.62:0/3846399123 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4700a5c60 con 0x7fe4800a3410 2024-04-19T01:42:27.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 4 set_mon_vals no callback set 2024-04-19T01:42:27.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe48521d640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:27.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe48521d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:27.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe48521d640 10 monclient: dump: 2024-04-19T01:42:27.968 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:27.968 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.968 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.968 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.968 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:27.968 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.968 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:27.968 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:27.968 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:27.968 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 10 monclient: shutdown 2024-04-19T01:42:27.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:27.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 1 -- 172.21.15.62:0/3846399123 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4800a3410 msgr2=0x7fe4800a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 1 --2- 172.21.15.62:0/3846399123 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4800a3410 0x7fe4800a37f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe4800a10a0 tx=0x7fe47001c7f0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 1 -- 172.21.15.62:0/3846399123 shutdown_connections 2024-04-19T01:42:27.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 1 --2- 172.21.15.62:0/3846399123 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4800a2d50 0x7fe4800a4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 1 --2- 172.21.15.62:0/3846399123 >> v2:172.21.15.71:3300/0 conn(0x7fe4800a2900 0x7fe4800a46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 1 --2- 172.21.15.62:0/3846399123 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4800a3410 0x7fe4800a37f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 1 -- 172.21.15.62:0/3846399123 >> 172.21.15.62:0/3846399123 conn(0x7fe4800aa2d0 msgr2=0x7fe4800a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:27.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 1 -- 172.21.15.62:0/3846399123 shutdown_connections 2024-04-19T01:42:27.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 1 -- 172.21.15.62:0/3846399123 wait complete. 2024-04-19T01:42:27.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 10 monclient: build_initial_monmap 2024-04-19T01:42:27.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 10 monclient: monmap: 2024-04-19T01:42:27.969 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:27.969 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:27.969 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:27.969 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:27.969 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:27.970 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.970 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:27.970 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:27.970 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:27.970 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 1 Processor -- start 2024-04-19T01:42:27.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 1 -- start start 2024-04-19T01:42:27.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 10 monclient: init 2024-04-19T01:42:27.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:27.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-19T01:42:27.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4800a2900 0x7fe4801cf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 10 monclient(hunting): picked mon.noname-a con 0x7fe4800a2900 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:27.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4800a2d50 0x7fe48013d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 10 monclient(hunting): picked mon.noname-c con 0x7fe4800a2d50 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:27.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe4800a3410 0x7fe48013df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 10 monclient(hunting): picked mon.noname-b con 0x7fe4800a3410 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:27.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe480141370 con 0x7fe4800a2900 2024-04-19T01:42:27.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe4801411f0 con 0x7fe4800a3410 2024-04-19T01:42:27.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:27.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fe4801414f0 con 0x7fe4800a2d50 2024-04-19T01:42:27.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:27.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe488483640 10 monclient(hunting): authenticate will time out at 933.786938s 2024-04-19T01:42:27.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe485a1e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4800a2d50 0x7fe48013d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe48621f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4800a2900 0x7fe4801cf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe485a1e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4800a2d50 0x7fe48013d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60508/0 (socket says 172.21.15.62:60508) 2024-04-19T01:42:27.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe485a1e640 1 -- 172.21.15.62:0/2868761106 learned_addr learned my addr 172.21.15.62:0/2868761106 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:27.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe48621f640 10 monclient(hunting): get_auth_request con 0x7fe4800a2900 auth_method 0 2024-04-19T01:42:27.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe48621f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:27.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe48621f640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:27.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe48621f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:27.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe485a1e640 10 monclient(hunting): get_auth_request con 0x7fe4800a2d50 auth_method 0 2024-04-19T01:42:27.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe485a1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:27.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe485a1e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:27.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe485a1e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:27.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe485a1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:27.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe485a1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:27.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe485a1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:27.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe48621f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:27.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe48621f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:27.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe48621f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:27.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe485a1e640 10 monclient(hunting): handle_auth_done global_id 4304 payload 824 2024-04-19T01:42:27.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe485a1e640 1 -- 172.21.15.62:0/2868761106 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe4800a3410 msgr2=0x7fe48013df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe486a20640 1 --2- 172.21.15.62:0/2868761106 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe4800a3410 0x7fe48013df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe485a1e640 1 --2- 172.21.15.62:0/2868761106 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe4800a3410 0x7fe48013df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe485a1e640 1 -- 172.21.15.62:0/2868761106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4800a2900 msgr2=0x7fe4801cf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:27.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe485a1e640 1 --2- 172.21.15.62:0/2868761106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4800a2900 0x7fe4801cf8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:27.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe485a1e640 10 monclient: _finish_hunting 0 2024-04-19T01:42:27.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe485a1e640 1 monclient: found mon.noname-c 2024-04-19T01:42:27.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe485a1e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:27.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe485a1e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:42:27.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe485a1e640 1 -- 172.21.15.62:0/2868761106 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4800a01f0 con 0x7fe4800a2d50 2024-04-19T01:42:27.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.962+0000 7fe48621f640 1 --2- 172.21.15.62:0/2868761106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4800a2900 0x7fe4801cf8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:42:27.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.966+0000 7fe485a1e640 1 -- 172.21.15.62:0/2868761106 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4800a2d50 msgr2=0x7fe48013d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:27.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.966+0000 7fe485a1e640 1 --2- 172.21.15.62:0/2868761106 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4800a2d50 0x7fe48013d9c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe47400d830 tx=0x7fe47400dcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:27.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.966+0000 7fe47effd640 1 -- 172.21.15.62:0/2868761106 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47400b8d0 con 0x7fe4800a2d50 2024-04-19T01:42:27.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.966+0000 7fe47effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:27.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.966+0000 7fe47effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:27.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.966+0000 7fe47effd640 10 monclient: dump: 2024-04-19T01:42:27.975 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:27.975 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.976 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.976 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.976 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:27.976 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.976 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:27.976 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:27.976 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:27.976 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.966+0000 7fe47effd640 10 monclient: _finish_auth 0 2024-04-19T01:42:27.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.966+0000 7fe47effd640 10 monclient: _check_auth_tickets 2024-04-19T01:42:27.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.966+0000 7fe47effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:27.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.966+0000 7fe47effd640 1 -- 172.21.15.62:0/2868761106 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe474019070 con 0x7fe4800a2d50 2024-04-19T01:42:27.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.966+0000 7fe47effd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:27.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.966+0000 7fe47effd640 1 -- 172.21.15.62:0/2868761106 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47400bc70 con 0x7fe4800a2d50 2024-04-19T01:42:27.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.966+0000 7fe47effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:27.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.966+0000 7fe47effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:27.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.966+0000 7fe47effd640 10 monclient: dump: 2024-04-19T01:42:27.977 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:27.977 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:27.977 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.977 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:27.977 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:27.977 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:27.977 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:27.977 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:27.977 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:27.978 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:27.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.966+0000 7fe488483640 5 monclient: authenticate success, global_id 4304 2024-04-19T01:42:27.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.966+0000 7fe488483640 10 monclient: _renew_subs 2024-04-19T01:42:27.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.966+0000 7fe488483640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:27.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.966+0000 7fe488483640 1 -- 172.21.15.62:0/2868761106 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4800a3f80 con 0x7fe4800a2d50 2024-04-19T01:42:27.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.966+0000 7fe47ffff640 4 set_mon_vals no callback set 2024-04-19T01:42:27.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.966+0000 7fe488483640 10 monclient: _renew_subs 2024-04-19T01:42:27.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.966+0000 7fe488483640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:27.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.966+0000 7fe488483640 1 -- 172.21.15.62:0/2868761106 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe480127090 con 0x7fe4800a2d50 2024-04-19T01:42:27.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.966+0000 7fe488483640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:27.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.966+0000 7fe488483640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:27.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.966+0000 7fe488483640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:27.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.966+0000 7fe47effd640 1 -- 172.21.15.62:0/2868761106 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fe47401e460 con 0x7fe4800a2d50 2024-04-19T01:42:27.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.966+0000 7fe488483640 1 -- 172.21.15.62:0/2868761106 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4800692b0 con 0x7fe4800a2d50 2024-04-19T01:42:27.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.966+0000 7fe47effd640 1 --2- 172.21.15.62:0/2868761106 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fe46c036470 0x7fe46c038930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:27.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.966+0000 7fe47effd640 1 -- 172.21.15.62:0/2868761106 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4346+0+0 (secure 0 0 0) 0x7fe474049770 con 0x7fe4800a2d50 2024-04-19T01:42:27.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.966+0000 7fe48621f640 1 --2- 172.21.15.62:0/2868761106 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fe46c036470 0x7fe46c038930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:27.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.966+0000 7fe48621f640 10 monclient: get_auth_request con 0x7fe46c036470 auth_method 0 2024-04-19T01:42:27.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.970+0000 7fe48621f640 1 --2- 172.21.15.62:0/2868761106 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fe46c036470 0x7fe46c038930 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe480073940 tx=0x7fe4700a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:27.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.970+0000 7fe47effd640 1 -- 172.21.15.62:0/2868761106 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe47400be10 con 0x7fe4800a2d50 2024-04-19T01:42:27.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.970+0000 7fe47effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:27.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:27.970+0000 7fe47effd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:28.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.126+0000 7fe488483640 1 -- 172.21.15.62:0/2868761106 --> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe48009fc30 con 0x7fe46c036470 2024-04-19T01:42:28.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.130+0000 7fe47effd640 1 -- 172.21.15.62:0/2868761106 <== mgr.4108 v2:172.21.15.71:6824/2133533992 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33750 (secure 0 0 0) 0x7fe48009fc30 con 0x7fe46c036470 2024-04-19T01:42:28.131 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:42:28.131 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-04-19T01:42:28.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.130+0000 7fe488483640 1 -- 172.21.15.62:0/2868761106 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fe46c036470 msgr2=0x7fe46c038930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:28.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.130+0000 7fe488483640 1 --2- 172.21.15.62:0/2868761106 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fe46c036470 0x7fe46c038930 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe480073940 tx=0x7fe4700a7070 comp rx=0 tx=0).stop 2024-04-19T01:42:28.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.130+0000 7fe488483640 10 monclient: shutdown 2024-04-19T01:42:28.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.130+0000 7fe488483640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:28.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.130+0000 7fe488483640 1 -- 172.21.15.62:0/2868761106 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4800a2d50 msgr2=0x7fe48013d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:28.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.130+0000 7fe488483640 1 --2- 172.21.15.62:0/2868761106 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4800a2d50 0x7fe48013d9c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe47400d830 tx=0x7fe47400dcf0 comp rx=0 tx=0).stop 2024-04-19T01:42:28.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.130+0000 7fe488483640 1 -- 172.21.15.62:0/2868761106 shutdown_connections 2024-04-19T01:42:28.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.130+0000 7fe488483640 1 --2- 172.21.15.62:0/2868761106 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fe46c036470 0x7fe46c038930 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:28.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.130+0000 7fe488483640 1 --2- 172.21.15.62:0/2868761106 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe4800a2d50 0x7fe48013d9c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:28.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.130+0000 7fe488483640 1 --2- 172.21.15.62:0/2868761106 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe4800a3410 0x7fe48013df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:28.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.130+0000 7fe488483640 1 --2- 172.21.15.62:0/2868761106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4800a2900 0x7fe4801cf8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:28.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.134+0000 7fe488483640 1 -- 172.21.15.62:0/2868761106 >> 172.21.15.62:0/2868761106 conn(0x7fe4800aa2d0 msgr2=0x7fe4800a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:28.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.134+0000 7fe488483640 1 -- 172.21.15.62:0/2868761106 shutdown_connections 2024-04-19T01:42:28.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.134+0000 7fe488483640 1 -- 172.21.15.62:0/2868761106 wait complete. 2024-04-19T01:42:28.149 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-04-19T01:42:26.704787+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":101,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164112,"kb_used_data":1856,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066928,"statfs":{"total":579820584960,"available":579652534272,"internally_reserved":0,"allocated":1900544,"data_stored":1317428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.681345"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.163776+0000","last_change":"2024-04-19T01:42:24.162228+0000","last_active":"2024-04-19T01:42:24.163776+0000","last_peered":"2024-04-19T01:42:24.163776+0000","last_clean":"2024-04-19T01:42:24.163776+0000","last_became_active":"2024-04-19T01:42:24.161645+0000","last_became_peered":"2024-04-19T01:42:24.161645+0000","last_unstale":"2024-04-19T01:42:24.163776+0000","last_undegraded":"2024-04-19T01:42:24.163776+0000","last_fullsized":"2024-04-19T01:42:24.163776+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:08:44.489901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.141485+0000","last_change":"2024-04-19T01:42:24.132804+0000","last_active":"2024-04-19T01:42:24.141485+0000","last_peered":"2024-04-19T01:42:24.141485+0000","last_clean":"2024-04-19T01:42:24.141485+0000","last_became_active":"2024-04-19T01:42:24.131937+0000","last_became_peered":"2024-04-19T01:42:24.131937+0000","last_unstale":"2024-04-19T01:42:24.141485+0000","last_undegraded":"2024-04-19T01:42:24.141485+0000","last_fullsized":"2024-04-19T01:42:24.141485+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T06:51:05.320754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.139253+0000","last_change":"2024-04-19T01:42:24.135590+0000","last_active":"2024-04-19T01:42:24.139253+0000","last_peered":"2024-04-19T01:42:24.139253+0000","last_clean":"2024-04-19T01:42:24.139253+0000","last_became_active":"2024-04-19T01:42:24.134918+0000","last_became_peered":"2024-04-19T01:42:24.134918+0000","last_unstale":"2024-04-19T01:42:24.139253+0000","last_undegraded":"2024-04-19T01:42:24.139253+0000","last_fullsized":"2024-04-19T01:42:24.139253+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T08:09:26.534746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.141458+0000","last_change":"2024-04-19T01:42:24.132575+0000","last_active":"2024-04-19T01:42:24.141458+0000","last_peered":"2024-04-19T01:42:24.141458+0000","last_clean":"2024-04-19T01:42:24.141458+0000","last_became_active":"2024-04-19T01:42:24.131894+0000","last_became_peered":"2024-04-19T01:42:24.131894+0000","last_unstale":"2024-04-19T01:42:24.141458+0000","last_undegraded":"2024-04-19T01:42:24.141458+0000","last_fullsized":"2024-04-19T01:42:24.141458+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:25:51.912215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.148595+0000","last_change":"2024-04-19T01:42:24.145699+0000","last_active":"2024-04-19T01:42:24.148595+0000","last_peered":"2024-04-19T01:42:24.148595+0000","last_clean":"2024-04-19T01:42:24.148595+0000","last_became_active":"2024-04-19T01:42:24.144746+0000","last_became_peered":"2024-04-19T01:42:24.144746+0000","last_unstale":"2024-04-19T01:42:24.148595+0000","last_undegraded":"2024-04-19T01:42:24.148595+0000","last_fullsized":"2024-04-19T01:42:24.148595+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T04:20:46.381920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.145097+0000","last_change":"2024-04-19T01:42:24.142754+0000","last_active":"2024-04-19T01:42:24.145097+0000","last_peered":"2024-04-19T01:42:24.145097+0000","last_clean":"2024-04-19T01:42:24.145097+0000","last_became_active":"2024-04-19T01:42:24.142309+0000","last_became_peered":"2024-04-19T01:42:24.142309+0000","last_unstale":"2024-04-19T01:42:24.145097+0000","last_undegraded":"2024-04-19T01:42:24.145097+0000","last_fullsized":"2024-04-19T01:42:24.145097+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:18:58.995723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.136907+0000","last_change":"2024-04-19T01:42:24.134484+0000","last_active":"2024-04-19T01:42:24.136907+0000","last_peered":"2024-04-19T01:42:24.136907+0000","last_clean":"2024-04-19T01:42:24.136907+0000","last_became_active":"2024-04-19T01:42:24.133282+0000","last_became_peered":"2024-04-19T01:42:24.133282+0000","last_unstale":"2024-04-19T01:42:24.136907+0000","last_undegraded":"2024-04-19T01:42:24.136907+0000","last_fullsized":"2024-04-19T01:42:24.136907+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T06:02:03.349002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.149236+0000","last_change":"2024-04-19T01:42:24.146550+0000","last_active":"2024-04-19T01:42:24.149236+0000","last_peered":"2024-04-19T01:42:24.149236+0000","last_clean":"2024-04-19T01:42:24.149236+0000","last_became_active":"2024-04-19T01:42:24.145667+0000","last_became_peered":"2024-04-19T01:42:24.145667+0000","last_unstale":"2024-04-19T01:42:24.149236+0000","last_undegraded":"2024-04-19T01:42:24.149236+0000","last_fullsized":"2024-04-19T01:42:24.149236+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:48:51.752906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":165,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.130442+0000","last_change":"2024-04-19T01:42:22.132251+0000","last_active":"2024-04-19T01:42:24.130442+0000","last_peered":"2024-04-19T01:42:24.130442+0000","last_clean":"2024-04-19T01:42:24.130442+0000","last_became_active":"2024-04-19T01:42:22.131432+0000","last_became_peered":"2024-04-19T01:42:22.131432+0000","last_unstale":"2024-04-19T01:42:24.130442+0000","last_undegraded":"2024-04-19T01:42:24.130442+0000","last_fullsized":"2024-04-19T01:42:24.130442+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:21.102542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:21.102542+0000","last_clean_scrub_stamp":"2024-04-19T01:42:21.102542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T06:47:12.046147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":101,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":101,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607554,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":139264,"data_stored":43262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":139264,"data_stored":43262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344164,"statfs":{"total":96636764160,"available":96608423936,"internally_reserved":0,"allocated":671744,"data_stored":572190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":139264,"data_stored":43262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":139264,"data_stored":43262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":671744,"data_stored":572190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:42:28.149 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-19T01:42:28.150 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-19T01:42:28.150 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-19T01:42:28.150 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-19T01:42:28.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e9782640 10 monclient: get_monmap_and_config 2024-04-19T01:42:28.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e9782640 10 monclient: build_initial_monmap 2024-04-19T01:42:28.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e9782640 10 monclient: monmap: 2024-04-19T01:42:28.267 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:28.267 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:28.267 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:28.271157+0000 2024-04-19T01:42:28.267 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:28.271157+0000 2024-04-19T01:42:28.267 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:28.267 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:28.267 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:28.267 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:28.267 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:28.267 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:28.268 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:28.268 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:28.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e9782640 1 Processor -- start 2024-04-19T01:42:28.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e9782640 1 -- start start 2024-04-19T01:42:28.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e9782640 10 monclient: init 2024-04-19T01:42:28.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e9782640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:28.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e9782640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-04-19T01:42:28.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e9782640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4e40a47e0 0x7fa4e40a4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:28.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e9782640 10 monclient(hunting): picked mon.noname-b con 0x7fa4e40a47e0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:28.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e9782640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa4e40a5100 addr v1:172.21.15.71:6789/0 2024-04-19T01:42:28.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e9782640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa4e41357c0 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:28.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e9782640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4e409f7c0 con 0x7fa4e41357c0 2024-04-19T01:42:28.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e9782640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4e409c2d0 con 0x7fa4e40a5100 2024-04-19T01:42:28.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e9782640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:28.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e9782640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fa4e409bf00 con 0x7fa4e40a47e0 2024-04-19T01:42:28.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e9782640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:28.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e9782640 10 monclient(hunting): authenticate will time out at 934.090782s 2024-04-19T01:42:28.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e2ffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4e40a47e0 0x7fa4e40a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:28.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e37fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fa4e41357c0 0x7fa4e413d600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:33608/0 (socket says 172.21.15.62:33608) 2024-04-19T01:42:28.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e37fe640 1 -- 172.21.15.62:0/3876638055 learned_addr learned my addr 172.21.15.62:0/3876638055 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:42:28.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e2ffd640 10 monclient(hunting): get_auth_request con 0x7fa4e40a47e0 auth_method 0 2024-04-19T01:42:28.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:28.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:28.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:28.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:28.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:28.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:28.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e2ffd640 10 monclient(hunting): handle_auth_done global_id 4307 payload 274 2024-04-19T01:42:28.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e2ffd640 1 -- 172.21.15.62:0/3876638055 >> v1:172.21.15.71:6789/0 conn(0x7fa4e40a5100 legacy=0x7fa4e41350b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:28.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e2ffd640 1 -- 172.21.15.62:0/3876638055 >> v1:172.21.15.62:6789/0 conn(0x7fa4e41357c0 legacy=0x7fa4e413d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:28.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e2ffd640 10 monclient: _finish_hunting 0 2024-04-19T01:42:28.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e2ffd640 1 monclient: found mon.noname-b 2024-04-19T01:42:28.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:28.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:42:28.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.266+0000 7fa4e2ffd640 1 -- 172.21.15.62:0/3876638055 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4e409faa0 con 0x7fa4e40a47e0 2024-04-19T01:42:28.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.270+0000 7fa4e2ffd640 1 -- 172.21.15.62:0/3876638055 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4e40a47e0 msgr2=0x7fa4e40a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:28.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.270+0000 7fa4e2ffd640 1 --2- 172.21.15.62:0/3876638055 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4e40a47e0 0x7fa4e40a4bc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc009dc0 tx=0x7fa4dc01c700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e45be8b50abcdb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:28.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.270+0000 7fa4e1ffb640 1 -- 172.21.15.62:0/3876638055 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc0a8740 con 0x7fa4e40a47e0 2024-04-19T01:42:28.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.270+0000 7fa4e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:28.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.270+0000 7fa4e1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:42:28.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.270+0000 7fa4e1ffb640 10 monclient: dump: 2024-04-19T01:42:28.273 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:28.273 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:28.273 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:28.273 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:28.274 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:28.274 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:28.274 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:28.274 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:28.274 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:28.274 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:28.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.270+0000 7fa4e1ffb640 10 monclient: _finish_auth 0 2024-04-19T01:42:28.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.270+0000 7fa4e1ffb640 10 monclient: _check_auth_tickets 2024-04-19T01:42:28.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.270+0000 7fa4e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:28.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.270+0000 7fa4e1ffb640 1 -- 172.21.15.62:0/3876638055 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4dc0a88e0 con 0x7fa4e40a47e0 2024-04-19T01:42:28.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.270+0000 7fa4e1ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:28.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.270+0000 7fa4e1ffb640 1 -- 172.21.15.62:0/3876638055 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc0a8cb0 con 0x7fa4e40a47e0 2024-04-19T01:42:28.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.270+0000 7fa4e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:28.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.270+0000 7fa4e1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:28.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.270+0000 7fa4e1ffb640 10 monclient: dump: 2024-04-19T01:42:28.275 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:28.275 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:28.275 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:28.275 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:28.275 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:28.275 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:28.275 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:28.275 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:28.275 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:28.275 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:28.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.270+0000 7fa4e9782640 5 monclient: authenticate success, global_id 4307 2024-04-19T01:42:28.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.270+0000 7fa4e9782640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:28.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.270+0000 7fa4e9782640 4 set_mon_vals no callback set 2024-04-19T01:42:28.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e9782640 10 monclient: shutdown 2024-04-19T01:42:28.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e9782640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:28.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e9782640 1 -- 172.21.15.62:0/3876638055 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4e40a47e0 msgr2=0x7fa4e40a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:28.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e9782640 1 --2- 172.21.15.62:0/3876638055 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4e40a47e0 0x7fa4e40a4bc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc009dc0 tx=0x7fa4dc01c700 comp rx=0 tx=0).stop 2024-04-19T01:42:28.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e9782640 1 -- 172.21.15.62:0/3876638055 shutdown_connections 2024-04-19T01:42:28.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e9782640 1 --2- 172.21.15.62:0/3876638055 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4e40a47e0 0x7fa4e40a4bc0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:28.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e9782640 1 -- 172.21.15.62:0/3876638055 >> 172.21.15.62:0/3876638055 conn(0x7fa4e40a23d0 msgr2=0x7fa4e40a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:28.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e9782640 1 -- 172.21.15.62:0/3876638055 shutdown_connections 2024-04-19T01:42:28.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e9782640 1 -- 172.21.15.62:0/3876638055 wait complete. 2024-04-19T01:42:28.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e9782640 10 monclient: build_initial_monmap 2024-04-19T01:42:28.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e9782640 10 monclient: monmap: 2024-04-19T01:42:28.280 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:28.280 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:28.280 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:28.280 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:28.281 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:28.281 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:28.281 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:28.281 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:28.281 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:28.281 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:28.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e9782640 1 Processor -- start 2024-04-19T01:42:28.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e9782640 1 -- start start 2024-04-19T01:42:28.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e9782640 10 monclient: init 2024-04-19T01:42:28.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e9782640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:28.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e9782640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-19T01:42:28.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e9782640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4e40a47e0 0x7fa4e412f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:28.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e9782640 10 monclient(hunting): picked mon.noname-a con 0x7fa4e40a47e0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:28.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e9782640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4e40a5100 0x7fa4e4136e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:28.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e9782640 10 monclient(hunting): picked mon.noname-c con 0x7fa4e40a5100 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:28.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e9782640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4e41357c0 0x7fa4e4137370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:28.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e9782640 10 monclient(hunting): picked mon.noname-b con 0x7fa4e41357c0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:28.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e9782640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:28.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e9782640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e409c8e0 con 0x7fa4e40a47e0 2024-04-19T01:42:28.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e9782640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:28.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e9782640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e409c760 con 0x7fa4e41357c0 2024-04-19T01:42:28.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e9782640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:28.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e9782640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fa4e4141540 con 0x7fa4e40a5100 2024-04-19T01:42:28.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e9782640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:28.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e9782640 10 monclient(hunting): authenticate will time out at 934.097879s 2024-04-19T01:42:28.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e27fc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4e40a5100 0x7fa4e4136e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:28.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e2ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4e40a47e0 0x7fa4e412f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:28.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e2ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4e40a47e0 0x7fa4e412f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36430/0 (socket says 172.21.15.62:36430) 2024-04-19T01:42:28.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e2ffd640 1 -- 172.21.15.62:0/1153111465 learned_addr learned my addr 172.21.15.62:0/1153111465 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:28.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e2ffd640 10 monclient(hunting): get_auth_request con 0x7fa4e40a47e0 auth_method 0 2024-04-19T01:42:28.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:28.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:28.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:28.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e27fc640 10 monclient(hunting): get_auth_request con 0x7fa4e40a5100 auth_method 0 2024-04-19T01:42:28.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:28.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e27fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:28.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:28.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:28.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:28.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:28.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:28.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:28.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:28.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e37fe640 1 --2- 172.21.15.62:0/1153111465 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4e41357c0 0x7fa4e4137370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:28.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e2ffd640 10 monclient(hunting): handle_auth_done global_id 4305 payload 824 2024-04-19T01:42:28.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e2ffd640 1 -- 172.21.15.62:0/1153111465 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4e40a5100 msgr2=0x7fa4e4136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:28.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e2ffd640 1 --2- 172.21.15.62:0/1153111465 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4e40a5100 0x7fa4e4136e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:28.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e2ffd640 1 -- 172.21.15.62:0/1153111465 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4e41357c0 msgr2=0x7fa4e4137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:28.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e2ffd640 1 --2- 172.21.15.62:0/1153111465 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4e41357c0 0x7fa4e4137370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:28.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e2ffd640 10 monclient: _finish_hunting 0 2024-04-19T01:42:28.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e2ffd640 1 monclient: found mon.noname-a 2024-04-19T01:42:28.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:28.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:28.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e2ffd640 1 -- 172.21.15.62:0/1153111465 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4e413a480 con 0x7fa4e40a47e0 2024-04-19T01:42:28.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e27fc640 1 --2- 172.21.15.62:0/1153111465 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4e40a5100 0x7fa4e4136e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:42:28.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e37fe640 10 monclient: get_auth_request con 0x7fa4e41357c0 auth_method 0 2024-04-19T01:42:28.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.274+0000 7fa4e37fe640 1 --2- 172.21.15.62:0/1153111465 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4e41357c0 0x7fa4e4137370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:42:28.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.278+0000 7fa4e2ffd640 1 -- 172.21.15.62:0/1153111465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4e40a47e0 msgr2=0x7fa4e412f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:28.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.278+0000 7fa4e2ffd640 1 --2- 172.21.15.62:0/1153111465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4e40a47e0 0x7fa4e412f770 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc01c530 tx=0x7fa4dc0194a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:28.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.278+0000 7fa4db7fe640 1 -- 172.21.15.62:0/1153111465 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc0b0a20 con 0x7fa4e40a47e0 2024-04-19T01:42:28.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.278+0000 7fa4db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:28.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.278+0000 7fa4db7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:28.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.278+0000 7fa4db7fe640 10 monclient: dump: 2024-04-19T01:42:28.285 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:28.285 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:28.285 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:28.285 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:28.285 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:28.285 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:28.285 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:28.285 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:28.285 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:28.285 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:28.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.278+0000 7fa4db7fe640 10 monclient: _finish_auth 0 2024-04-19T01:42:28.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.278+0000 7fa4db7fe640 10 monclient: _check_auth_tickets 2024-04-19T01:42:28.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.278+0000 7fa4db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:28.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.278+0000 7fa4db7fe640 1 -- 172.21.15.62:0/1153111465 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4dc0bf070 con 0x7fa4e40a47e0 2024-04-19T01:42:28.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.278+0000 7fa4e9782640 5 monclient: authenticate success, global_id 4305 2024-04-19T01:42:28.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.278+0000 7fa4e9782640 10 monclient: _renew_subs 2024-04-19T01:42:28.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.278+0000 7fa4e9782640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:28.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.278+0000 7fa4e9782640 1 -- 172.21.15.62:0/1153111465 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4e413a760 con 0x7fa4e40a47e0 2024-04-19T01:42:28.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.278+0000 7fa4db7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:28.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.278+0000 7fa4db7fe640 1 -- 172.21.15.62:0/1153111465 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc0b0de0 con 0x7fa4e40a47e0 2024-04-19T01:42:28.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.278+0000 7fa4db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:28.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.278+0000 7fa4db7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:28.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.278+0000 7fa4db7fe640 10 monclient: dump: 2024-04-19T01:42:28.286 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:28.286 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:28.286 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:28.287 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:28.287 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:28.287 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:28.287 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:28.287 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:28.287 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:28.287 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:28.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.278+0000 7fa4e17fa640 4 set_mon_vals no callback set 2024-04-19T01:42:28.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.278+0000 7fa4e9782640 10 monclient: _renew_subs 2024-04-19T01:42:28.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.278+0000 7fa4e9782640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:28.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.278+0000 7fa4e9782640 1 -- 172.21.15.62:0/1153111465 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4e413dbd0 con 0x7fa4e40a47e0 2024-04-19T01:42:28.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.278+0000 7fa4e9782640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:28.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.278+0000 7fa4e9782640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:28.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.278+0000 7fa4e9782640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:28.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.278+0000 7fa4e9782640 1 -- 172.21.15.62:0/1153111465 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4e413ae20 con 0x7fa4e40a47e0 2024-04-19T01:42:28.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.278+0000 7fa4db7fe640 1 -- 172.21.15.62:0/1153111465 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fa4dc0c4460 con 0x7fa4e40a47e0 2024-04-19T01:42:28.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.278+0000 7fa4db7fe640 1 --2- 172.21.15.62:0/1153111465 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa4c4036470 0x7fa4c4038930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:28.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.278+0000 7fa4db7fe640 1 -- 172.21.15.62:0/1153111465 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4346+0+0 (secure 0 0 0) 0x7fa4dc0e4740 con 0x7fa4e40a47e0 2024-04-19T01:42:28.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.278+0000 7fa4e27fc640 1 --2- 172.21.15.62:0/1153111465 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa4c4036470 0x7fa4c4038930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:28.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.278+0000 7fa4e27fc640 10 monclient: get_auth_request con 0x7fa4c4036470 auth_method 0 2024-04-19T01:42:28.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.282+0000 7fa4e27fc640 1 --2- 172.21.15.62:0/1153111465 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa4c4036470 0x7fa4c4038930 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa4e4138630 tx=0x7fa4cc007e40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:28.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.282+0000 7fa4db7fe640 1 -- 172.21.15.62:0/1153111465 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa4e413ae20 con 0x7fa4e40a47e0 2024-04-19T01:42:28.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.282+0000 7fa4db7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:28.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.282+0000 7fa4db7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:28.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.482+0000 7fa4e9782640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-04-19T01:42:28.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.482+0000 7fa4e9782640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-04-19T01:42:28.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.482+0000 7fa4e9782640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:28.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.482+0000 7fa4e9782640 1 -- 172.21.15.62:0/1153111465 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fa4e40692b0 con 0x7fa4e40a47e0 2024-04-19T01:42:28.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.482+0000 7fa4db7fe640 1 -- 172.21.15.62:0/1153111465 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fa4e413a480 con 0x7fa4e40a47e0 2024-04-19T01:42:28.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.482+0000 7fa4db7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-04-19T01:42:28.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.482+0000 7fa4db7fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:28.485 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:42:28.486 INFO:teuthology.orchestra.run.smithi062.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-19T01:42:28.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.486+0000 7fa4e9782640 1 -- 172.21.15.62:0/1153111465 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa4c4036470 msgr2=0x7fa4c4038930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:28.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.486+0000 7fa4e9782640 1 --2- 172.21.15.62:0/1153111465 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa4c4036470 0x7fa4c4038930 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa4e4138630 tx=0x7fa4cc007e40 comp rx=0 tx=0).stop 2024-04-19T01:42:28.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.486+0000 7fa4e9782640 10 monclient: shutdown 2024-04-19T01:42:28.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.486+0000 7fa4e9782640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:28.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.486+0000 7fa4e9782640 1 -- 172.21.15.62:0/1153111465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4e40a47e0 msgr2=0x7fa4e412f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:28.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.486+0000 7fa4e9782640 1 --2- 172.21.15.62:0/1153111465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4e40a47e0 0x7fa4e412f770 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc01c530 tx=0x7fa4dc0194a0 comp rx=0 tx=0).stop 2024-04-19T01:42:28.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.486+0000 7fa4e9782640 1 -- 172.21.15.62:0/1153111465 shutdown_connections 2024-04-19T01:42:28.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.486+0000 7fa4e9782640 1 --2- 172.21.15.62:0/1153111465 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa4c4036470 0x7fa4c4038930 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:28.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.486+0000 7fa4e9782640 1 --2- 172.21.15.62:0/1153111465 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4e40a5100 0x7fa4e4136e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:28.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.486+0000 7fa4e9782640 1 --2- 172.21.15.62:0/1153111465 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4e41357c0 0x7fa4e4137370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:28.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.486+0000 7fa4e9782640 1 --2- 172.21.15.62:0/1153111465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4e40a47e0 0x7fa4e412f770 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:28.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.486+0000 7fa4e9782640 1 -- 172.21.15.62:0/1153111465 >> 172.21.15.62:0/1153111465 conn(0x7fa4e40a23d0 msgr2=0x7fa4e40a11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:28.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.486+0000 7fa4e9782640 1 -- 172.21.15.62:0/1153111465 shutdown_connections 2024-04-19T01:42:28.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.486+0000 7fa4e9782640 1 -- 172.21.15.62:0/1153111465 wait complete. 2024-04-19T01:42:28.505 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-19T01:42:28.505 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-04-19T01:42:28.517 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-04-19T01:42:28.517 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-19T01:42:28.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2ca080640 10 monclient: get_monmap_and_config 2024-04-19T01:42:28.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2ca080640 10 monclient: build_initial_monmap 2024-04-19T01:42:28.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2ca080640 10 monclient: monmap: 2024-04-19T01:42:28.639 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:28.639 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:28.639 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:28.642992+0000 2024-04-19T01:42:28.639 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:28.642992+0000 2024-04-19T01:42:28.639 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:28.639 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:28.639 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:28.639 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:28.639 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:28.639 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:28.639 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:28.639 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:28.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2ca080640 1 Processor -- start 2024-04-19T01:42:28.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2ca080640 1 -- start start 2024-04-19T01:42:28.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2ca080640 10 monclient: init 2024-04-19T01:42:28.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2ca080640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:28.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2ca080640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-04-19T01:42:28.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2ca080640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe2c40a47e0 0x7fe2c409f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:28.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2ca080640 10 monclient(hunting): picked mon.noname-c con 0x7fe2c40a47e0 addr v2:172.21.15.71:3300/0 2024-04-19T01:42:28.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2ca080640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe2c40a4bc0 addr v1:172.21.15.71:6789/0 2024-04-19T01:42:28.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2ca080640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe2c4139560 0x7fe2c413b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:28.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2ca080640 10 monclient(hunting): picked mon.noname-a con 0x7fe2c4139560 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:28.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2ca080640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2c409e6e0 con 0x7fe2c40a4bc0 2024-04-19T01:42:28.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2ca080640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:28.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2ca080640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fe2c409d4e0 con 0x7fe2c4139560 2024-04-19T01:42:28.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2ca080640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:28.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2ca080640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fe2c409e9c0 con 0x7fe2c40a47e0 2024-04-19T01:42:28.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2ca080640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:28.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2ca080640 10 monclient(hunting): authenticate will time out at 934.462421s 2024-04-19T01:42:28.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2c3fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe2c4139560 0x7fe2c413b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:28.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2c3fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe2c4139560 0x7fe2c413b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36442/0 (socket says 172.21.15.62:36442) 2024-04-19T01:42:28.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2c3fff640 1 -- 172.21.15.62:0/2567502415 learned_addr learned my addr 172.21.15.62:0/2567502415 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:28.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2c3fff640 10 monclient(hunting): get_auth_request con 0x7fe2c4139560 auth_method 0 2024-04-19T01:42:28.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:28.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2c3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:28.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:28.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:28.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:28.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:28.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2c37fe640 1 --2- 172.21.15.62:0/2567502415 >> v2:172.21.15.71:3300/0 conn(0x7fe2c40a47e0 0x7fe2c409f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:28.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2c37fe640 10 monclient(hunting): get_auth_request con 0x7fe2c40a47e0 auth_method 0 2024-04-19T01:42:28.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:28.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2c37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:28.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:28.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2c3fff640 10 monclient(hunting): handle_auth_done global_id 4308 payload 274 2024-04-19T01:42:28.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2c3fff640 1 -- 172.21.15.62:0/2567502415 >> v2:172.21.15.71:3300/0 conn(0x7fe2c40a47e0 msgr2=0x7fe2c409f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:28.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2c3fff640 1 --2- 172.21.15.62:0/2567502415 >> v2:172.21.15.71:3300/0 conn(0x7fe2c40a47e0 0x7fe2c409f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:28.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2c3fff640 1 -- 172.21.15.62:0/2567502415 >> v1:172.21.15.71:6789/0 conn(0x7fe2c40a4bc0 legacy=0x7fe2c40a4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:28.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2c3fff640 10 monclient: _finish_hunting 0 2024-04-19T01:42:28.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2c3fff640 1 monclient: found mon.noname-a 2024-04-19T01:42:28.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:28.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2c3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:28.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.638+0000 7fe2c3fff640 1 -- 172.21.15.62:0/2567502415 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2c409f990 con 0x7fe2c4139560 2024-04-19T01:42:28.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.642+0000 7fe2c3fff640 1 -- 172.21.15.62:0/2567502415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2c4139560 msgr2=0x7fe2c413b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:28.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.642+0000 7fe2c3fff640 1 --2- 172.21.15.62:0/2567502415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2c4139560 0x7fe2c413b950 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe2c409fc20 tx=0x7fe2bc01eb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df3afff6380c9639 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:28.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.642+0000 7fe2c27fc640 1 -- 172.21.15.62:0/2567502415 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc00b2d0 con 0x7fe2c4139560 2024-04-19T01:42:28.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.642+0000 7fe2c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:28.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.642+0000 7fe2c27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:28.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.642+0000 7fe2c27fc640 10 monclient: dump: 2024-04-19T01:42:28.647 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:28.647 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:28.647 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:28.647 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:28.647 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:28.647 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:28.647 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:28.647 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:28.647 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:28.647 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:28.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.642+0000 7fe2c27fc640 10 monclient: _finish_auth 0 2024-04-19T01:42:28.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.642+0000 7fe2c27fc640 10 monclient: _check_auth_tickets 2024-04-19T01:42:28.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.642+0000 7fe2c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:28.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.642+0000 7fe2c27fc640 1 -- 172.21.15.62:0/2567502415 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2bc00b470 con 0x7fe2c4139560 2024-04-19T01:42:28.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.642+0000 7fe2c27fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:28.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.642+0000 7fe2c27fc640 1 -- 172.21.15.62:0/2567502415 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc0a78f0 con 0x7fe2c4139560 2024-04-19T01:42:28.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.642+0000 7fe2c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:28.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.642+0000 7fe2c27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:28.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.642+0000 7fe2c27fc640 10 monclient: dump: 2024-04-19T01:42:28.648 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:28.648 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:28.648 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:28.648 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:28.648 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:28.648 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:28.648 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:28.648 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:28.649 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:28.649 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:28.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.642+0000 7fe2ca080640 5 monclient: authenticate success, global_id 4308 2024-04-19T01:42:28.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.642+0000 7fe2ca080640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:28.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.642+0000 7fe2ca080640 4 set_mon_vals no callback set 2024-04-19T01:42:28.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.642+0000 7fe2ca080640 10 monclient: shutdown 2024-04-19T01:42:28.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.642+0000 7fe2ca080640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:28.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.642+0000 7fe2ca080640 1 -- 172.21.15.62:0/2567502415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2c4139560 msgr2=0x7fe2c413b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:28.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.642+0000 7fe2ca080640 1 --2- 172.21.15.62:0/2567502415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2c4139560 0x7fe2c413b950 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe2c409fc20 tx=0x7fe2bc01eb40 comp rx=0 tx=0).stop 2024-04-19T01:42:28.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2ca080640 1 -- 172.21.15.62:0/2567502415 shutdown_connections 2024-04-19T01:42:28.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2ca080640 1 --2- 172.21.15.62:0/2567502415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2c4139560 0x7fe2c413b950 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:28.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2ca080640 1 --2- 172.21.15.62:0/2567502415 >> v2:172.21.15.71:3300/0 conn(0x7fe2c40a47e0 0x7fe2c409f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:28.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2ca080640 1 -- 172.21.15.62:0/2567502415 >> 172.21.15.62:0/2567502415 conn(0x7fe2c40a23d0 msgr2=0x7fe2c40a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:28.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2ca080640 1 -- 172.21.15.62:0/2567502415 shutdown_connections 2024-04-19T01:42:28.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2ca080640 1 -- 172.21.15.62:0/2567502415 wait complete. 2024-04-19T01:42:28.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2ca080640 10 monclient: build_initial_monmap 2024-04-19T01:42:28.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2ca080640 10 monclient: monmap: 2024-04-19T01:42:28.650 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:28.650 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:28.650 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:28.650 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:28.650 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:28.650 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:28.650 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:28.650 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:28.651 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:28.651 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:28.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2ca080640 1 Processor -- start 2024-04-19T01:42:28.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2ca080640 1 -- start start 2024-04-19T01:42:28.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2ca080640 10 monclient: init 2024-04-19T01:42:28.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2ca080640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:28.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2ca080640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-19T01:42:28.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2ca080640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2c40a47e0 0x7fe2c41cf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:28.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2ca080640 10 monclient(hunting): picked mon.noname-b con 0x7fe2c40a47e0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:28.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2ca080640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2c40a4bc0 0x7fe2c4135310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:28.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2ca080640 10 monclient(hunting): picked mon.noname-c con 0x7fe2c40a4bc0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:28.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2ca080640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2c4139560 0x7fe2c4135850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:28.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2ca080640 10 monclient(hunting): picked mon.noname-a con 0x7fe2c4139560 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:28.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2ca080640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:28.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2ca080640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe2c409c610 con 0x7fe2c4139560 2024-04-19T01:42:28.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2ca080640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:28.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2ca080640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe2c409c490 con 0x7fe2c40a47e0 2024-04-19T01:42:28.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2ca080640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:28.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2ca080640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fe2c409c790 con 0x7fe2c40a4bc0 2024-04-19T01:42:28.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2ca080640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:28.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2ca080640 10 monclient(hunting): authenticate will time out at 934.469529s 2024-04-19T01:42:28.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c2ffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2c40a4bc0 0x7fe2c4135310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:28.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2c4139560 0x7fe2c4135850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:28.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c2ffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2c40a4bc0 0x7fe2c4135310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60536/0 (socket says 172.21.15.62:60536) 2024-04-19T01:42:28.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c2ffd640 1 -- 172.21.15.62:0/1056603620 learned_addr learned my addr 172.21.15.62:0/1056603620 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:28.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2c4139560 0x7fe2c4135850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36450/0 (socket says 172.21.15.62:36450) 2024-04-19T01:42:28.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c2ffd640 10 monclient(hunting): get_auth_request con 0x7fe2c40a4bc0 auth_method 0 2024-04-19T01:42:28.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:28.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:28.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:28.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c3fff640 10 monclient(hunting): get_auth_request con 0x7fe2c4139560 auth_method 0 2024-04-19T01:42:28.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:28.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:28.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:28.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:28.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:28.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:28.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:28.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:28.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:28.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c37fe640 1 --2- 172.21.15.62:0/1056603620 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2c40a47e0 0x7fe2c41cf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:28.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c37fe640 10 monclient(hunting): get_auth_request con 0x7fe2c40a47e0 auth_method 0 2024-04-19T01:42:28.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:28.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:28.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:28.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c2ffd640 10 monclient(hunting): handle_auth_done global_id 4313 payload 824 2024-04-19T01:42:28.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c2ffd640 1 -- 172.21.15.62:0/1056603620 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2c40a47e0 msgr2=0x7fe2c41cf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:28.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c2ffd640 1 --2- 172.21.15.62:0/1056603620 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2c40a47e0 0x7fe2c41cf2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:28.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c2ffd640 1 -- 172.21.15.62:0/1056603620 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2c4139560 msgr2=0x7fe2c4135850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:28.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c2ffd640 1 --2- 172.21.15.62:0/1056603620 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2c4139560 0x7fe2c4135850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:28.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c2ffd640 10 monclient: _finish_hunting 0 2024-04-19T01:42:28.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c2ffd640 1 monclient: found mon.noname-c 2024-04-19T01:42:28.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:28.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:42:28.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c2ffd640 1 -- 172.21.15.62:0/1056603620 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2c409e050 con 0x7fe2c40a4bc0 2024-04-19T01:42:28.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.646+0000 7fe2c3fff640 1 --2- 172.21.15.62:0/1056603620 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2c4139560 0x7fe2c4135850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:42:28.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.650+0000 7fe2c2ffd640 1 -- 172.21.15.62:0/1056603620 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2c40a4bc0 msgr2=0x7fe2c4135310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:28.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.650+0000 7fe2c2ffd640 1 --2- 172.21.15.62:0/1056603620 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2c40a4bc0 0x7fe2c4135310 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe2b800c480 tx=0x7fe2b800c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:28.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.650+0000 7fe2abfff640 1 -- 172.21.15.62:0/1056603620 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b8012a40 con 0x7fe2c40a4bc0 2024-04-19T01:42:28.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.650+0000 7fe2abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:28.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.650+0000 7fe2abfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:28.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.650+0000 7fe2abfff640 10 monclient: dump: 2024-04-19T01:42:28.660 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:28.660 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:28.660 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:28.660 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:28.660 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:28.660 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:28.661 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:28.661 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:28.661 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:28.661 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:28.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.650+0000 7fe2abfff640 10 monclient: _finish_auth 0 2024-04-19T01:42:28.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.650+0000 7fe2abfff640 10 monclient: _check_auth_tickets 2024-04-19T01:42:28.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.650+0000 7fe2abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:28.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.650+0000 7fe2ca080640 5 monclient: authenticate success, global_id 4313 2024-04-19T01:42:28.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.650+0000 7fe2ca080640 10 monclient: _renew_subs 2024-04-19T01:42:28.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.650+0000 7fe2abfff640 1 -- 172.21.15.62:0/1056603620 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2b8017070 con 0x7fe2c40a4bc0 2024-04-19T01:42:28.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.650+0000 7fe2ca080640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:28.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.650+0000 7fe2ca080640 1 -- 172.21.15.62:0/1056603620 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2c4123ce0 con 0x7fe2c40a4bc0 2024-04-19T01:42:28.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.650+0000 7fe2abfff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:28.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.650+0000 7fe2abfff640 1 -- 172.21.15.62:0/1056603620 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b8012e40 con 0x7fe2c40a4bc0 2024-04-19T01:42:28.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.650+0000 7fe2abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:28.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.650+0000 7fe2abfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:28.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.650+0000 7fe2abfff640 10 monclient: dump: 2024-04-19T01:42:28.662 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:28.662 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:28.662 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:28.662 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:28.662 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:28.662 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:28.662 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:28.662 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:28.662 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:28.663 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:28.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.650+0000 7fe2ca080640 10 monclient: _renew_subs 2024-04-19T01:42:28.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.650+0000 7fe2c1ffb640 4 set_mon_vals no callback set 2024-04-19T01:42:28.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.650+0000 7fe2ca080640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:28.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.650+0000 7fe2ca080640 1 -- 172.21.15.62:0/1056603620 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2c4123440 con 0x7fe2c40a4bc0 2024-04-19T01:42:28.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.650+0000 7fe2ca080640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:28.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.650+0000 7fe2ca080640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:28.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.650+0000 7fe2ca080640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:28.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.650+0000 7fe2ca080640 1 -- 172.21.15.62:0/1056603620 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2c40692b0 con 0x7fe2c40a4bc0 2024-04-19T01:42:28.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.650+0000 7fe2abfff640 1 -- 172.21.15.62:0/1056603620 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fe2b801c460 con 0x7fe2c40a4bc0 2024-04-19T01:42:28.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.650+0000 7fe2abfff640 1 --2- 172.21.15.62:0/1056603620 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fe2b0036420 0x7fe2b00388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:28.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.650+0000 7fe2abfff640 1 -- 172.21.15.62:0/1056603620 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4346+0+0 (secure 0 0 0) 0x7fe2b80490f0 con 0x7fe2c40a4bc0 2024-04-19T01:42:28.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.650+0000 7fe2c37fe640 1 --2- 172.21.15.62:0/1056603620 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fe2b0036420 0x7fe2b00388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:28.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.650+0000 7fe2c37fe640 10 monclient: get_auth_request con 0x7fe2b0036420 auth_method 0 2024-04-19T01:42:28.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.654+0000 7fe2c37fe640 1 --2- 172.21.15.62:0/1056603620 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fe2b0036420 0x7fe2b00388e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe2b400b9d0 tx=0x7fe2b4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:28.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.654+0000 7fe2abfff640 1 -- 172.21.15.62:0/1056603620 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe2c409e050 con 0x7fe2c40a4bc0 2024-04-19T01:42:28.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.654+0000 7fe2abfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:28.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.654+0000 7fe2abfff640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:28.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.810+0000 7fe2ca080640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-19T01:42:28.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.810+0000 7fe2ca080640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-19T01:42:28.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.810+0000 7fe2ca080640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:28.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.810+0000 7fe2ca080640 1 -- 172.21.15.62:0/1056603620 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe2c413b880 con 0x7fe2c40a4bc0 2024-04-19T01:42:28.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.814+0000 7fe2abfff640 1 -- 172.21.15.62:0/1056603620 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12197 (secure 0 0 0) 0x7fe2b801a760 con 0x7fe2c40a4bc0 2024-04-19T01:42:28.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.814+0000 7fe2abfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-19T01:42:28.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.814+0000 7fe2abfff640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:28.816 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:42:28.816 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":16,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","created":"2024-04-19T01:42:09.115311+0000","modified":"2024-04-19T01:42:25.142305+0000","last_up_change":"2024-04-19T01:42:21.102542+0000","last_in_change":"2024-04-19T01:42:12.761455+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-19T01:42:20.748429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-19T01:42:22.720005+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"64304e60-9fe6-413b-8559-3b8ebd179a7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":3870805292},{"type":"v1","addr":"172.21.15.62:6802","nonce":3870805292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":3870805292},{"type":"v1","addr":"172.21.15.62:6806","nonce":3870805292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":3870805292},{"type":"v1","addr":"172.21.15.62:6814","nonce":3870805292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":3870805292},{"type":"v1","addr":"172.21.15.62:6810","nonce":3870805292}]},"public_addr":"172.21.15.62:6802/3870805292","cluster_addr":"172.21.15.62:6806/3870805292","heartbeat_back_addr":"172.21.15.62:6814/3870805292","heartbeat_front_addr":"172.21.15.62:6810/3870805292","state":["exists","up"]},{"osd":1,"uuid":"dec72cc0-fa6d-4fe9-b9cd-19dec591ab68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":199109455},{"type":"v1","addr":"172.21.15.62:6817","nonce":199109455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":199109455},{"type":"v1","addr":"172.21.15.62:6819","nonce":199109455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":199109455},{"type":"v1","addr":"172.21.15.62:6823","nonce":199109455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":199109455},{"type":"v1","addr":"172.21.15.62:6821","nonce":199109455}]},"public_addr":"172.21.15.62:6817/199109455","cluster_addr":"172.21.15.62:6819/199109455","heartbeat_back_addr":"172.21.15.62:6823/199109455","heartbeat_front_addr":"172.21.15.62:6821/199109455","state":["exists","up"]},{"osd":2,"uuid":"2045906a-e0fd-4c41-b73c-27bf7103243b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6801","nonce":1128256055},{"type":"v1","addr":"172.21.15.62:6803","nonce":1128256055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6805","nonce":1128256055},{"type":"v1","addr":"172.21.15.62:6807","nonce":1128256055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6813","nonce":1128256055},{"type":"v1","addr":"172.21.15.62:6815","nonce":1128256055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6809","nonce":1128256055},{"type":"v1","addr":"172.21.15.62:6811","nonce":1128256055}]},"public_addr":"172.21.15.62:6803/1128256055","cluster_addr":"172.21.15.62:6807/1128256055","heartbeat_back_addr":"172.21.15.62:6815/1128256055","heartbeat_front_addr":"172.21.15.62:6811/1128256055","state":["exists","up"]},{"osd":3,"uuid":"230f6e15-548c-4338-a8ec-067b8f85bd2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":2632024688},{"type":"v1","addr":"172.21.15.71:6817","nonce":2632024688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":2632024688},{"type":"v1","addr":"172.21.15.71:6819","nonce":2632024688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":2632024688},{"type":"v1","addr":"172.21.15.71:6823","nonce":2632024688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":2632024688},{"type":"v1","addr":"172.21.15.71:6821","nonce":2632024688}]},"public_addr":"172.21.15.71:6817/2632024688","cluster_addr":"172.21.15.71:6819/2632024688","heartbeat_back_addr":"172.21.15.71:6823/2632024688","heartbeat_front_addr":"172.21.15.71:6821/2632024688","state":["exists","up"]},{"osd":4,"uuid":"06789be2-244f-4b6e-ada2-30b6771676af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":3342848064},{"type":"v1","addr":"172.21.15.71:6809","nonce":3342848064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3342848064},{"type":"v1","addr":"172.21.15.71:6811","nonce":3342848064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":3342848064},{"type":"v1","addr":"172.21.15.71:6815","nonce":3342848064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":3342848064},{"type":"v1","addr":"172.21.15.71:6813","nonce":3342848064}]},"public_addr":"172.21.15.71:6809/3342848064","cluster_addr":"172.21.15.71:6811/3342848064","heartbeat_back_addr":"172.21.15.71:6815/3342848064","heartbeat_front_addr":"172.21.15.71:6813/3342848064","state":["exists","up"]},{"osd":5,"uuid":"552be339-732f-4801-b14d-3d9b08e7f1f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":196330660},{"type":"v1","addr":"172.21.15.71:6801","nonce":196330660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":196330660},{"type":"v1","addr":"172.21.15.71:6803","nonce":196330660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":196330660},{"type":"v1","addr":"172.21.15.71:6807","nonce":196330660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":196330660},{"type":"v1","addr":"172.21.15.71:6805","nonce":196330660}]},"public_addr":"172.21.15.71:6801/196330660","cluster_addr":"172.21.15.71:6803/196330660","heartbeat_back_addr":"172.21.15.71:6807/196330660","heartbeat_front_addr":"172.21.15.71:6805/196330660","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:42:18.255067+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:42:18.295998+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:42:18.304269+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:42:18.274604+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:42:18.338084+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:42:18.149308+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T01:42:28.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.814+0000 7fe2ca080640 1 -- 172.21.15.62:0/1056603620 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fe2b0036420 msgr2=0x7fe2b00388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:28.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.814+0000 7fe2ca080640 1 --2- 172.21.15.62:0/1056603620 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fe2b0036420 0x7fe2b00388e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe2b400b9d0 tx=0x7fe2b4008040 comp rx=0 tx=0).stop 2024-04-19T01:42:28.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.814+0000 7fe2ca080640 10 monclient: shutdown 2024-04-19T01:42:28.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.814+0000 7fe2ca080640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:28.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.814+0000 7fe2ca080640 1 -- 172.21.15.62:0/1056603620 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2c40a4bc0 msgr2=0x7fe2c4135310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:28.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.814+0000 7fe2ca080640 1 --2- 172.21.15.62:0/1056603620 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2c40a4bc0 0x7fe2c4135310 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe2b800c480 tx=0x7fe2b800c940 comp rx=0 tx=0).stop 2024-04-19T01:42:28.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.814+0000 7fe2ca080640 1 -- 172.21.15.62:0/1056603620 shutdown_connections 2024-04-19T01:42:28.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.814+0000 7fe2ca080640 1 --2- 172.21.15.62:0/1056603620 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fe2b0036420 0x7fe2b00388e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:28.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.814+0000 7fe2ca080640 1 --2- 172.21.15.62:0/1056603620 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe2c40a4bc0 0x7fe2c4135310 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:28.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.814+0000 7fe2ca080640 1 --2- 172.21.15.62:0/1056603620 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2c4139560 0x7fe2c4135850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:28.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.814+0000 7fe2ca080640 1 --2- 172.21.15.62:0/1056603620 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2c40a47e0 0x7fe2c41cf2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:28.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.814+0000 7fe2ca080640 1 -- 172.21.15.62:0/1056603620 >> 172.21.15.62:0/1056603620 conn(0x7fe2c40a23d0 msgr2=0x7fe2c40a11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:28.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.814+0000 7fe2ca080640 1 -- 172.21.15.62:0/1056603620 shutdown_connections 2024-04-19T01:42:28.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.814+0000 7fe2ca080640 1 -- 172.21.15.62:0/1056603620 wait complete. 2024-04-19T01:42:28.833 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-19T01:42:20.748429+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-19T01:42:22.720005+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-19T01:42:28.834 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-19T01:42:28.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b0a366640 10 monclient: get_monmap_and_config 2024-04-19T01:42:28.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b0a366640 10 monclient: build_initial_monmap 2024-04-19T01:42:28.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b0a366640 10 monclient: monmap: 2024-04-19T01:42:28.964 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:28.964 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:28.964 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:28.968176+0000 2024-04-19T01:42:28.964 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:28.968176+0000 2024-04-19T01:42:28.964 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:28.964 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:28.964 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:28.965 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:28.965 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:28.965 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:28.965 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:28.965 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:28.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b0a366640 1 Processor -- start 2024-04-19T01:42:28.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b0a366640 1 -- start start 2024-04-19T01:42:28.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b0a366640 10 monclient: init 2024-04-19T01:42:28.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b0a366640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:28.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b0a366640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-04-19T01:42:28.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b0a366640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3b040a47e0 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:28.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b0a366640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3b040a52d0 0x7f3b0413a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b0a366640 10 monclient(hunting): picked mon.noname-a con 0x7f3b040a52d0 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b0a366640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3b0413ab20 0x7f3b0413cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b0a366640 10 monclient(hunting): picked mon.noname-b con 0x7f3b0413ab20 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b0a366640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b0409f7c0 con 0x7f3b040a47e0 2024-04-19T01:42:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b0a366640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b0a366640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f3b0409bf00 con 0x7f3b040a52d0 2024-04-19T01:42:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b0a366640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b0a366640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f3b0409ee90 con 0x7f3b0413ab20 2024-04-19T01:42:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b0a366640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b0a366640 10 monclient(hunting): authenticate will time out at 934.787600s 2024-04-19T01:42:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b08903640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3b0413ab20 0x7f3b0413cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b037fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3b040a52d0 0x7f3b0413a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b08903640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3b0413ab20 0x7f3b0413cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60548/0 (socket says 172.21.15.62:60548) 2024-04-19T01:42:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b08903640 1 -- 172.21.15.62:0/2339926124 learned_addr learned my addr 172.21.15.62:0/2339926124 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b08903640 10 monclient(hunting): get_auth_request con 0x7f3b0413ab20 auth_method 0 2024-04-19T01:42:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b08903640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b08903640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b08903640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b037fe640 10 monclient(hunting): get_auth_request con 0x7f3b040a52d0 auth_method 0 2024-04-19T01:42:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b037fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b037fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b08903640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b08903640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.962+0000 7f3b08903640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.966+0000 7f3b08903640 10 monclient(hunting): handle_auth_done global_id 4316 payload 274 2024-04-19T01:42:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.966+0000 7f3b08903640 1 -- 172.21.15.62:0/2339926124 >> v2:172.21.15.62:3300/0 conn(0x7f3b040a52d0 msgr2=0x7f3b0413a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.966+0000 7f3b08903640 1 --2- 172.21.15.62:0/2339926124 >> v2:172.21.15.62:3300/0 conn(0x7f3b040a52d0 0x7f3b0413a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.966+0000 7f3b08903640 1 -- 172.21.15.62:0/2339926124 >> v1:172.21.15.62:6789/0 conn(0x7f3b040a47e0 legacy=0x7f3b040a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.966+0000 7f3b08903640 10 monclient: _finish_hunting 0 2024-04-19T01:42:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.966+0000 7f3b08903640 1 monclient: found mon.noname-b 2024-04-19T01:42:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.966+0000 7f3b08903640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.966+0000 7f3b08903640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:42:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.966+0000 7f3b08903640 1 -- 172.21.15.62:0/2339926124 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b0409c2d0 con 0x7f3b0413ab20 2024-04-19T01:42:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.966+0000 7f3b037fe640 1 --2- 172.21.15.62:0/2339926124 >> v2:172.21.15.62:3300/0 conn(0x7f3b040a52d0 0x7f3b0413a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:42:28.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b08903640 1 -- 172.21.15.62:0/2339926124 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3b0413ab20 msgr2=0x7f3b0413cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:28.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b08903640 1 --2- 172.21.15.62:0/2339926124 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3b0413ab20 0x7f3b0413cf10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3afc009e80 tx=0x7f3afc01c780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbd3a0c79676a643 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b02ffd640 1 -- 172.21.15.62:0/2339926124 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3afc0ac070 con 0x7f3b0413ab20 2024-04-19T01:42:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b02ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:42:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b02ffd640 10 monclient: dump: 2024-04-19T01:42:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:28.972 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b02ffd640 10 monclient: _finish_auth 0 2024-04-19T01:42:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b02ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:42:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b02ffd640 1 -- 172.21.15.62:0/2339926124 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3afc0a8430 con 0x7f3b0413ab20 2024-04-19T01:42:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b02ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 5 monclient: authenticate success, global_id 4316 2024-04-19T01:42:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b02ffd640 1 -- 172.21.15.62:0/2339926124 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3afc0a8800 con 0x7f3b0413ab20 2024-04-19T01:42:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 4 set_mon_vals no callback set 2024-04-19T01:42:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b02ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b02ffd640 10 monclient: dump: 2024-04-19T01:42:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:28.974 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:28.974 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:28.974 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:28.974 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:28.974 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:28.974 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:28.974 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:28.974 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:28.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 10 monclient: shutdown 2024-04-19T01:42:28.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:28.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 1 -- 172.21.15.62:0/2339926124 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3b0413ab20 msgr2=0x7f3b0413cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:28.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 1 --2- 172.21.15.62:0/2339926124 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3b0413ab20 0x7f3b0413cf10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3afc009e80 tx=0x7f3afc01c780 comp rx=0 tx=0).stop 2024-04-19T01:42:28.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 1 -- 172.21.15.62:0/2339926124 shutdown_connections 2024-04-19T01:42:28.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 1 --2- 172.21.15.62:0/2339926124 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3b0413ab20 0x7f3b0413cf10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:28.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 1 --2- 172.21.15.62:0/2339926124 >> v2:172.21.15.62:3300/0 conn(0x7f3b040a52d0 0x7f3b0413a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:28.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 1 -- 172.21.15.62:0/2339926124 >> 172.21.15.62:0/2339926124 conn(0x7f3b040a23d0 msgr2=0x7f3b040a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:28.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 1 -- 172.21.15.62:0/2339926124 shutdown_connections 2024-04-19T01:42:28.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 1 -- 172.21.15.62:0/2339926124 wait complete. 2024-04-19T01:42:28.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 10 monclient: build_initial_monmap 2024-04-19T01:42:28.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 10 monclient: monmap: 2024-04-19T01:42:28.979 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:28.979 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:28.979 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:28.979 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:28.979 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:28.979 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:28.980 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 1 Processor -- start 2024-04-19T01:42:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 1 -- start start 2024-04-19T01:42:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 10 monclient: init 2024-04-19T01:42:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-19T01:42:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b040a47e0 0x7f3b041cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 10 monclient(hunting): picked mon.noname-a con 0x7f3b040a47e0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b040a52d0 0x7f3b041352a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 10 monclient(hunting): picked mon.noname-b con 0x7f3b040a52d0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3b0413ab20 0x7f3b0411bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 10 monclient(hunting): picked mon.noname-c con 0x7f3b0413ab20 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3b0413fc70 con 0x7f3b040a47e0 2024-04-19T01:42:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f3b0409c970 con 0x7f3b040a52d0 2024-04-19T01:42:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f3b0413fdf0 con 0x7f3b0413ab20 2024-04-19T01:42:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b0a366640 10 monclient(hunting): authenticate will time out at 934.795640s 2024-04-19T01:42:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b03fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b040a47e0 0x7f3b041cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b08903640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3b0413ab20 0x7f3b0411bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b08903640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3b0413ab20 0x7f3b0411bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60556/0 (socket says 172.21.15.62:60556) 2024-04-19T01:42:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b08903640 1 -- 172.21.15.62:0/652756307 learned_addr learned my addr 172.21.15.62:0/652756307 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b03fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b040a47e0 0x7f3b041cd1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36468/0 (socket says 172.21.15.62:36468) 2024-04-19T01:42:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b08903640 10 monclient(hunting): get_auth_request con 0x7f3b0413ab20 auth_method 0 2024-04-19T01:42:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b08903640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b08903640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b08903640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b03fff640 10 monclient(hunting): get_auth_request con 0x7f3b040a47e0 auth_method 0 2024-04-19T01:42:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b03fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b03fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b08903640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b08903640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b08903640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.970+0000 7f3b03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b037fe640 1 --2- 172.21.15.62:0/652756307 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b040a52d0 0x7f3b041352a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b037fe640 10 monclient(hunting): get_auth_request con 0x7f3b040a52d0 auth_method 0 2024-04-19T01:42:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b037fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b037fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b08903640 10 monclient(hunting): handle_auth_done global_id 4319 payload 824 2024-04-19T01:42:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b08903640 1 -- 172.21.15.62:0/652756307 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b040a52d0 msgr2=0x7f3b041352a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b08903640 1 --2- 172.21.15.62:0/652756307 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b040a52d0 0x7f3b041352a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b08903640 1 -- 172.21.15.62:0/652756307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b040a47e0 msgr2=0x7f3b041cd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b08903640 1 --2- 172.21.15.62:0/652756307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b040a47e0 0x7f3b041cd1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b08903640 10 monclient: _finish_hunting 0 2024-04-19T01:42:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b08903640 1 monclient: found mon.noname-c 2024-04-19T01:42:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b08903640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b08903640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:42:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b08903640 1 -- 172.21.15.62:0/652756307 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b040726d0 con 0x7f3b0413ab20 2024-04-19T01:42:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b03fff640 1 --2- 172.21.15.62:0/652756307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b040a47e0 0x7f3b041cd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:42:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b08903640 1 -- 172.21.15.62:0/652756307 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3b0413ab20 msgr2=0x7f3b0411bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b08903640 1 --2- 172.21.15.62:0/652756307 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3b0413ab20 0x7f3b0411bf20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3afc01d020 tx=0x7f3afc019520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b00ff9640 1 -- 172.21.15.62:0/652756307 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3afc0bf070 con 0x7f3b0413ab20 2024-04-19T01:42:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b00ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b00ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b00ff9640 10 monclient: dump: 2024-04-19T01:42:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:28.984 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b00ff9640 10 monclient: _finish_auth 0 2024-04-19T01:42:28.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b00ff9640 10 monclient: _check_auth_tickets 2024-04-19T01:42:28.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b00ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:28.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b00ff9640 1 -- 172.21.15.62:0/652756307 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3afc0b1450 con 0x7f3b0413ab20 2024-04-19T01:42:28.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b0a366640 5 monclient: authenticate success, global_id 4319 2024-04-19T01:42:28.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b0a366640 10 monclient: _renew_subs 2024-04-19T01:42:28.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b0a366640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:28.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b0a366640 1 -- 172.21.15.62:0/652756307 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b0411fc50 con 0x7f3b0413ab20 2024-04-19T01:42:28.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b00ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:28.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b00ff9640 1 -- 172.21.15.62:0/652756307 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3afc0ac040 con 0x7f3b0413ab20 2024-04-19T01:42:28.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b00ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:28.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b00ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:28.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b00ff9640 10 monclient: dump: 2024-04-19T01:42:28.985 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:28.985 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:28.985 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:28.985 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:28.986 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:28.986 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:28.986 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:28.986 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:28.986 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:28.986 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:28.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b0a366640 10 monclient: _renew_subs 2024-04-19T01:42:28.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b0a366640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:28.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b0a366640 1 -- 172.21.15.62:0/652756307 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b0411f970 con 0x7f3b0413ab20 2024-04-19T01:42:28.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b027fc640 4 set_mon_vals no callback set 2024-04-19T01:42:28.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b0a366640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:28.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b00ff9640 1 -- 172.21.15.62:0/652756307 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f3afc0af070 con 0x7f3b0413ab20 2024-04-19T01:42:28.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b0a366640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:28.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b0a366640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:28.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b0a366640 1 -- 172.21.15.62:0/652756307 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b040692b0 con 0x7f3b0413ab20 2024-04-19T01:42:28.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b00ff9640 1 --2- 172.21.15.62:0/652756307 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f3af0036450 0x7f3af0038910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:28.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.974+0000 7f3b00ff9640 1 -- 172.21.15.62:0/652756307 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4346+0+0 (secure 0 0 0) 0x7f3afc0e2f70 con 0x7f3b0413ab20 2024-04-19T01:42:28.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.978+0000 7f3b03fff640 1 --2- 172.21.15.62:0/652756307 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f3af0036450 0x7f3af0038910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:28.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.978+0000 7f3b03fff640 10 monclient: get_auth_request con 0x7f3af0036450 auth_method 0 2024-04-19T01:42:28.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.978+0000 7f3b03fff640 1 --2- 172.21.15.62:0/652756307 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f3af0036450 0x7f3af0038910 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3af4007fd0 tx=0x7f3af4007e40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:28.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.982+0000 7f3b00ff9640 1 -- 172.21.15.62:0/652756307 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3b040726d0 con 0x7f3b0413ab20 2024-04-19T01:42:28.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.982+0000 7f3b00ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:28.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:28.982+0000 7f3b00ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:29.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.142+0000 7f3b0a366640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-04-19T01:42:29.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.142+0000 7f3b0a366640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-04-19T01:42:29.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.142+0000 7f3b0a366640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:29.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.142+0000 7f3b0a366640 1 -- 172.21.15.62:0/652756307 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3b0411f410 con 0x7f3b0413ab20 2024-04-19T01:42:29.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.142+0000 7f3b00ff9640 1 -- 172.21.15.62:0/652756307 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7f3b0409f7c0 con 0x7f3b0413ab20 2024-04-19T01:42:29.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.142+0000 7f3b00ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-04-19T01:42:29.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.142+0000 7f3b00ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:29.143 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-04-19T01:42:29.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.142+0000 7f3b0a366640 1 -- 172.21.15.62:0/652756307 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f3af0036450 msgr2=0x7f3af0038910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:29.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.142+0000 7f3b0a366640 1 --2- 172.21.15.62:0/652756307 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f3af0036450 0x7f3af0038910 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3af4007fd0 tx=0x7f3af4007e40 comp rx=0 tx=0).stop 2024-04-19T01:42:29.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.142+0000 7f3b0a366640 10 monclient: shutdown 2024-04-19T01:42:29.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.142+0000 7f3b0a366640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:29.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.142+0000 7f3b0a366640 1 -- 172.21.15.62:0/652756307 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3b0413ab20 msgr2=0x7f3b0411bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:29.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.142+0000 7f3b0a366640 1 --2- 172.21.15.62:0/652756307 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3b0413ab20 0x7f3b0411bf20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3afc01d020 tx=0x7f3afc019520 comp rx=0 tx=0).stop 2024-04-19T01:42:29.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.142+0000 7f3b0a366640 1 -- 172.21.15.62:0/652756307 shutdown_connections 2024-04-19T01:42:29.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.142+0000 7f3b0a366640 1 --2- 172.21.15.62:0/652756307 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f3af0036450 0x7f3af0038910 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:29.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.142+0000 7f3b0a366640 1 --2- 172.21.15.62:0/652756307 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f3b0413ab20 0x7f3b0411bf20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:29.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.142+0000 7f3b0a366640 1 --2- 172.21.15.62:0/652756307 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b040a52d0 0x7f3b041352a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:29.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.142+0000 7f3b0a366640 1 --2- 172.21.15.62:0/652756307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3b040a47e0 0x7f3b041cd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:29.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.142+0000 7f3b0a366640 1 -- 172.21.15.62:0/652756307 >> 172.21.15.62:0/652756307 conn(0x7f3b040a23d0 msgr2=0x7f3b0413b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:29.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.142+0000 7f3b0a366640 1 -- 172.21.15.62:0/652756307 shutdown_connections 2024-04-19T01:42:29.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.142+0000 7f3b0a366640 1 -- 172.21.15.62:0/652756307 wait complete. 2024-04-19T01:42:29.162 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-04-19T01:42:29.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa14f36640 10 monclient: get_monmap_and_config 2024-04-19T01:42:29.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa14f36640 10 monclient: build_initial_monmap 2024-04-19T01:42:29.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa14f36640 10 monclient: monmap: 2024-04-19T01:42:29.282 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:29.282 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:29.282 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:29.286627+0000 2024-04-19T01:42:29.282 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:29.286627+0000 2024-04-19T01:42:29.282 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:29.283 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:29.283 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:29.283 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:29.283 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:29.283 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:29.283 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:29.283 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:29.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa14f36640 1 Processor -- start 2024-04-19T01:42:29.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa14f36640 1 -- start start 2024-04-19T01:42:29.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa14f36640 10 monclient: init 2024-04-19T01:42:29.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa14f36640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:29.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa14f36640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-04-19T01:42:29.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa14f36640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7faa100a45d0 0x7faa1009c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:29.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa14f36640 10 monclient(hunting): picked mon.noname-c con 0x7faa100a45d0 addr v2:172.21.15.71:3300/0 2024-04-19T01:42:29.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa14f36640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faa100a49b0 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:29.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa14f36640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faa1013c8c0 addr v1:172.21.15.71:6789/0 2024-04-19T01:42:29.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa14f36640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa100a7590 con 0x7faa100a49b0 2024-04-19T01:42:29.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa14f36640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa1009bc80 con 0x7faa1013c8c0 2024-04-19T01:42:29.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa14f36640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:29.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa14f36640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7faa1009c240 con 0x7faa100a45d0 2024-04-19T01:42:29.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa14f36640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:29.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa14f36640 10 monclient(hunting): authenticate will time out at 935.106247s 2024-04-19T01:42:29.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa0dd9b640 1 -- >> v1:172.21.15.62:6789/0 conn(0x7faa100a49b0 legacy=0x7faa100a4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:42:29.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa0dd9b640 1 -- >> v1:172.21.15.62:6789/0 conn(0x7faa100a49b0 legacy=0x7faa100a4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:42:29.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa0dd9b640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7faa100a49b0 0x7faa100a4dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify read banner and identify addresses failed 2024-04-19T01:42:29.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa0e59c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7faa100a45d0 0x7faa1009c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:29.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa0e59c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7faa100a45d0 0x7faa1009c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.62:41450/0 (socket says 172.21.15.62:41450) 2024-04-19T01:42:29.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa0e59c640 1 -- 172.21.15.62:0/2332580680 learned_addr learned my addr 172.21.15.62:0/2332580680 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:29.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa0e59c640 10 monclient(hunting): get_auth_request con 0x7faa100a45d0 auth_method 0 2024-04-19T01:42:29.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:29.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa0e59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:29.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa0e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:29.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:29.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:29.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:29.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa0e59c640 10 monclient(hunting): handle_auth_done global_id 4288 payload 274 2024-04-19T01:42:29.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa0e59c640 1 -- 172.21.15.62:0/2332580680 >> v1:172.21.15.71:6789/0 conn(0x7faa1013c8c0 legacy=0x7faa1013ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:29.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa0e59c640 1 -- 172.21.15.62:0/2332580680 >> v1:172.21.15.62:6789/0 conn(0x7faa100a49b0 legacy=0x7faa100a4dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:42:29.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa0e59c640 10 monclient: _finish_hunting 0 2024-04-19T01:42:29.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa0e59c640 1 monclient: found mon.noname-c 2024-04-19T01:42:29.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa0e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:29.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa0e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-04-19T01:42:29.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.282+0000 7faa0e59c640 1 -- 172.21.15.62:0/2332580680 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa10073210 con 0x7faa100a45d0 2024-04-19T01:42:29.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0e59c640 1 -- 172.21.15.62:0/2332580680 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa100a45d0 msgr2=0x7faa1009c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:29.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0e59c640 1 --2- 172.21.15.62:0/2332580680 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa100a45d0 0x7faa1009c940 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7faa00009870 tx=0x7faa0001c420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87f610da37c40c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:29.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0d59a640 1 -- 172.21.15.62:0/2332580680 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa000b0070 con 0x7faa100a45d0 2024-04-19T01:42:29.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:29.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:29.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0d59a640 10 monclient: dump: 2024-04-19T01:42:29.290 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:29.290 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:29.290 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:29.290 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:29.290 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:29.290 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:29.290 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:29.291 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:29.291 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:29.291 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:29.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0d59a640 10 monclient: _finish_auth 0 2024-04-19T01:42:29.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0d59a640 10 monclient: _check_auth_tickets 2024-04-19T01:42:29.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:29.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0d59a640 1 -- 172.21.15.62:0/2332580680 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa000ab600 con 0x7faa100a45d0 2024-04-19T01:42:29.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0d59a640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:29.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0d59a640 1 -- 172.21.15.62:0/2332580680 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa000ab9d0 con 0x7faa100a45d0 2024-04-19T01:42:29.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:29.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:42:29.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0d59a640 10 monclient: dump: 2024-04-19T01:42:29.291 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:29.291 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:29.291 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:29.292 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:29.292 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:29.292 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:29.292 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:29.292 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:29.292 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:29.292 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:29.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 5 monclient: authenticate success, global_id 4288 2024-04-19T01:42:29.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:29.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 4 set_mon_vals no callback set 2024-04-19T01:42:29.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 10 monclient: shutdown 2024-04-19T01:42:29.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:29.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 1 -- 172.21.15.62:0/2332580680 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa100a45d0 msgr2=0x7faa1009c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:29.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 1 --2- 172.21.15.62:0/2332580680 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa100a45d0 0x7faa1009c940 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7faa00009870 tx=0x7faa0001c420 comp rx=0 tx=0).stop 2024-04-19T01:42:29.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 1 -- 172.21.15.62:0/2332580680 shutdown_connections 2024-04-19T01:42:29.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 1 --2- 172.21.15.62:0/2332580680 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa100a45d0 0x7faa1009c940 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:29.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 1 -- 172.21.15.62:0/2332580680 >> 172.21.15.62:0/2332580680 conn(0x7faa100aa1d0 msgr2=0x7faa100a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:29.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 1 -- 172.21.15.62:0/2332580680 shutdown_connections 2024-04-19T01:42:29.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 1 -- 172.21.15.62:0/2332580680 wait complete. 2024-04-19T01:42:29.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 10 monclient: build_initial_monmap 2024-04-19T01:42:29.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 10 monclient: monmap: 2024-04-19T01:42:29.296 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:29.296 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:29.296 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:29.296 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:29.296 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:29.296 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:29.296 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:29.296 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:29.296 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:29.296 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:29.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 1 Processor -- start 2024-04-19T01:42:29.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 1 -- start start 2024-04-19T01:42:29.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 10 monclient: init 2024-04-19T01:42:29.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:29.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-19T01:42:29.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faa100a45d0 0x7faa10137250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:29.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 10 monclient(hunting): picked mon.noname-c con 0x7faa100a45d0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:29.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa100a49b0 0x7faa1009fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:29.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 10 monclient(hunting): picked mon.noname-b con 0x7faa100a49b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:29.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa1013c8c0 0x7faa100a0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:29.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 10 monclient(hunting): picked mon.noname-a con 0x7faa1013c8c0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:29.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:29.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7faa10142f80 con 0x7faa1013c8c0 2024-04-19T01:42:29.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:29.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7faa10142e00 con 0x7faa100a49b0 2024-04-19T01:42:29.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:29.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7faa10143100 con 0x7faa100a45d0 2024-04-19T01:42:29.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:29.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa14f36640 10 monclient(hunting): authenticate will time out at 935.115566s 2024-04-19T01:42:29.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0e59c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faa100a45d0 0x7faa10137250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:29.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0e59c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faa100a45d0 0x7faa10137250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60562/0 (socket says 172.21.15.62:60562) 2024-04-19T01:42:29.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0e59c640 1 -- 172.21.15.62:0/2078919746 learned_addr learned my addr 172.21.15.62:0/2078919746 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:29.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0e59c640 10 monclient(hunting): get_auth_request con 0x7faa100a45d0 auth_method 0 2024-04-19T01:42:29.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:29.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0e59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:29.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:29.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0ed9d640 1 --2- 172.21.15.62:0/2078919746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa1013c8c0 0x7faa100a0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:29.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:29.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:29.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:29.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0ed9d640 10 monclient(hunting): get_auth_request con 0x7faa1013c8c0 auth_method 0 2024-04-19T01:42:29.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:29.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:29.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:29.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0dd9b640 1 --2- 172.21.15.62:0/2078919746 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa100a49b0 0x7faa1009fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:29.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:29.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:29.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:29.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0dd9b640 10 monclient(hunting): get_auth_request con 0x7faa100a49b0 auth_method 0 2024-04-19T01:42:29.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:29.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:29.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.290+0000 7faa0dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:29.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7faa0e59c640 10 monclient(hunting): handle_auth_done global_id 4322 payload 824 2024-04-19T01:42:29.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7faa0e59c640 1 -- 172.21.15.62:0/2078919746 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa100a49b0 msgr2=0x7faa1009fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:29.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7faa0e59c640 1 --2- 172.21.15.62:0/2078919746 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa100a49b0 0x7faa1009fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:29.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7faa0e59c640 1 -- 172.21.15.62:0/2078919746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa1013c8c0 msgr2=0x7faa100a0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:29.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7faa0e59c640 1 --2- 172.21.15.62:0/2078919746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa1013c8c0 0x7faa100a0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:29.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7faa0e59c640 10 monclient: _finish_hunting 0 2024-04-19T01:42:29.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7faa0e59c640 1 monclient: found mon.noname-c 2024-04-19T01:42:29.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7faa0e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:29.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7faa0e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:42:29.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7faa0e59c640 1 -- 172.21.15.62:0/2078919746 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa1013f8d0 con 0x7faa100a45d0 2024-04-19T01:42:29.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7faa0e59c640 1 -- 172.21.15.62:0/2078919746 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faa100a45d0 msgr2=0x7faa10137250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:29.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7faa0e59c640 1 --2- 172.21.15.62:0/2078919746 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faa100a45d0 0x7faa10137250 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7faa000ae040 tx=0x7faa0001ac60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:29.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7fa9feffd640 1 -- 172.21.15.62:0/2078919746 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa000ba020 con 0x7faa100a45d0 2024-04-19T01:42:29.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7fa9feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:29.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7fa9feffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:29.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7fa9feffd640 10 monclient: dump: 2024-04-19T01:42:29.302 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:29.303 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:29.303 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:29.303 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:29.303 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:29.303 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:29.303 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:29.303 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:29.303 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:29.303 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:29.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7fa9feffd640 10 monclient: _finish_auth 0 2024-04-19T01:42:29.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7fa9feffd640 10 monclient: _check_auth_tickets 2024-04-19T01:42:29.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7fa9feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:29.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7fa9feffd640 1 -- 172.21.15.62:0/2078919746 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa000abe10 con 0x7faa100a45d0 2024-04-19T01:42:29.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7fa9feffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:29.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7fa9feffd640 1 -- 172.21.15.62:0/2078919746 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa000b0040 con 0x7faa100a45d0 2024-04-19T01:42:29.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7faa14f36640 5 monclient: authenticate success, global_id 4322 2024-04-19T01:42:29.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7faa14f36640 10 monclient: _renew_subs 2024-04-19T01:42:29.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7faa14f36640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:29.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7faa0cd99640 4 set_mon_vals no callback set 2024-04-19T01:42:29.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7faa14f36640 1 -- 172.21.15.62:0/2078919746 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa10120910 con 0x7faa100a45d0 2024-04-19T01:42:29.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7fa9feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:29.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7fa9feffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:29.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7fa9feffd640 10 monclient: dump: 2024-04-19T01:42:29.304 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:29.304 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:29.304 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:29.304 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:29.304 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:29.304 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:29.304 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:29.304 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:29.305 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:29.305 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:29.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7faa14f36640 10 monclient: _renew_subs 2024-04-19T01:42:29.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7faa14f36640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:29.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7faa14f36640 1 -- 172.21.15.62:0/2078919746 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faa100a33f0 con 0x7faa100a45d0 2024-04-19T01:42:29.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7fa9feffd640 1 -- 172.21.15.62:0/2078919746 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7faa000bd020 con 0x7faa100a45d0 2024-04-19T01:42:29.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7faa14f36640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:29.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7faa14f36640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:29.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7faa14f36640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:29.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7faa14f36640 1 -- 172.21.15.62:0/2078919746 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa100692b0 con 0x7faa100a45d0 2024-04-19T01:42:29.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7fa9feffd640 1 --2- 172.21.15.62:0/2078919746 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa9f0036450 0x7fa9f0038910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:29.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.294+0000 7fa9feffd640 1 -- 172.21.15.62:0/2078919746 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4346+0+0 (secure 0 0 0) 0x7faa000e3620 con 0x7faa100a45d0 2024-04-19T01:42:29.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.298+0000 7faa0dd9b640 1 --2- 172.21.15.62:0/2078919746 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa9f0036450 0x7fa9f0038910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:29.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.298+0000 7faa0dd9b640 10 monclient: get_auth_request con 0x7fa9f0036450 auth_method 0 2024-04-19T01:42:29.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.298+0000 7faa0dd9b640 1 --2- 172.21.15.62:0/2078919746 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa9f0036450 0x7fa9f0038910 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7faa100a1360 tx=0x7faa04007ca0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:29.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.302+0000 7fa9feffd640 1 -- 172.21.15.62:0/2078919746 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7faa000c4070 con 0x7faa100a45d0 2024-04-19T01:42:29.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.302+0000 7fa9feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:29.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.302+0000 7fa9feffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:29.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.466+0000 7faa14f36640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-04-19T01:42:29.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.466+0000 7faa14f36640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-04-19T01:42:29.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.466+0000 7faa14f36640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:29.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.466+0000 7faa14f36640 1 -- 172.21.15.62:0/2078919746 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) -- 0x7faa10120350 con 0x7faa100a45d0 2024-04-19T01:42:29.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.470+0000 7fa9feffd640 1 -- 172.21.15.62:0/2078919746 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v16) ==== 92+0+10 (secure 0 0 0) 0x7faa1013f8d0 con 0x7faa100a45d0 2024-04-19T01:42:29.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.470+0000 7fa9feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-04-19T01:42:29.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.470+0000 7fa9feffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:29.470 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 8 2024-04-19T01:42:29.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.470+0000 7faa14f36640 1 -- 172.21.15.62:0/2078919746 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa9f0036450 msgr2=0x7fa9f0038910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:29.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.470+0000 7faa14f36640 1 --2- 172.21.15.62:0/2078919746 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa9f0036450 0x7fa9f0038910 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7faa100a1360 tx=0x7faa04007ca0 comp rx=0 tx=0).stop 2024-04-19T01:42:29.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.470+0000 7faa14f36640 10 monclient: shutdown 2024-04-19T01:42:29.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.470+0000 7faa14f36640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:29.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.470+0000 7faa14f36640 1 -- 172.21.15.62:0/2078919746 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faa100a45d0 msgr2=0x7faa10137250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:29.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.470+0000 7faa14f36640 1 --2- 172.21.15.62:0/2078919746 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faa100a45d0 0x7faa10137250 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7faa000ae040 tx=0x7faa0001ac60 comp rx=0 tx=0).stop 2024-04-19T01:42:29.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.470+0000 7faa14f36640 1 -- 172.21.15.62:0/2078919746 shutdown_connections 2024-04-19T01:42:29.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.470+0000 7faa14f36640 1 --2- 172.21.15.62:0/2078919746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa1013c8c0 0x7faa100a0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:29.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.470+0000 7faa14f36640 1 --2- 172.21.15.62:0/2078919746 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa100a49b0 0x7faa1009fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:29.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.470+0000 7faa14f36640 1 --2- 172.21.15.62:0/2078919746 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa9f0036450 0x7fa9f0038910 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:29.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.470+0000 7faa14f36640 1 --2- 172.21.15.62:0/2078919746 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faa100a45d0 0x7faa10137250 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:29.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.470+0000 7faa14f36640 1 -- 172.21.15.62:0/2078919746 >> 172.21.15.62:0/2078919746 conn(0x7faa100aa1d0 msgr2=0x7faa1013d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:29.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.470+0000 7faa14f36640 1 -- 172.21.15.62:0/2078919746 shutdown_connections 2024-04-19T01:42:29.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.470+0000 7faa14f36640 1 -- 172.21.15.62:0/2078919746 wait complete. 2024-04-19T01:42:29.489 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-04-19T01:42:29.489 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:42:29.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.602+0000 7fbf604c0640 10 monclient: get_monmap_and_config 2024-04-19T01:42:29.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.602+0000 7fbf604c0640 10 monclient: build_initial_monmap 2024-04-19T01:42:29.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.602+0000 7fbf604c0640 10 monclient: monmap: 2024-04-19T01:42:29.605 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:29.605 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:29.605 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:29.609147+0000 2024-04-19T01:42:29.605 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:29.609147+0000 2024-04-19T01:42:29.605 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:29.605 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:29.605 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:29.605 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:29.605 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:29.606 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:29.606 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:29.606 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:29.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.602+0000 7fbf604c0640 1 Processor -- start 2024-04-19T01:42:29.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.602+0000 7fbf604c0640 1 -- start start 2024-04-19T01:42:29.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.602+0000 7fbf604c0640 10 monclient: init 2024-04-19T01:42:29.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf604c0640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:29.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf604c0640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-04-19T01:42:29.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf604c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbf580a2ef0 addr v1:172.21.15.71:6789/0 2024-04-19T01:42:29.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf604c0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbf5813a570 0x7fbf5813c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:29.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf604c0640 10 monclient(hunting): picked mon.noname-b con 0x7fbf5813a570 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:29.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf604c0640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fbf5813cea0 0x7fbf5813f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:29.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf604c0640 10 monclient(hunting): picked mon.noname-c con 0x7fbf5813cea0 addr v2:172.21.15.71:3300/0 2024-04-19T01:42:29.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf604c0640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf58137550 con 0x7fbf580a2ef0 2024-04-19T01:42:29.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf604c0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:29.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf604c0640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fbf5809bd90 con 0x7fbf5813cea0 2024-04-19T01:42:29.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf604c0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:29.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf604c0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fbf5809c070 con 0x7fbf5813a570 2024-04-19T01:42:29.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf604c0640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:29.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf604c0640 10 monclient(hunting): authenticate will time out at 935.428607s 2024-04-19T01:42:29.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf5da5b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbf5813a570 0x7fbf5813c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:29.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf5da5b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbf5813a570 0x7fbf5813c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60570/0 (socket says 172.21.15.62:60570) 2024-04-19T01:42:29.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf5da5b640 1 -- 172.21.15.62:0/679606076 learned_addr learned my addr 172.21.15.62:0/679606076 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:29.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf5da5b640 10 monclient(hunting): get_auth_request con 0x7fbf5813a570 auth_method 0 2024-04-19T01:42:29.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf5da5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:29.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf5da5b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:29.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf5da5b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:29.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf5da5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:29.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf5da5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:29.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf5da5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:29.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf5da5b640 10 monclient(hunting): handle_auth_done global_id 4325 payload 274 2024-04-19T01:42:29.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf5da5b640 1 -- 172.21.15.62:0/679606076 >> v2:172.21.15.71:3300/0 conn(0x7fbf5813cea0 msgr2=0x7fbf5813f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:29.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf5ea5d640 1 --2- 172.21.15.62:0/679606076 >> v2:172.21.15.71:3300/0 conn(0x7fbf5813cea0 0x7fbf5813f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:29.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf5da5b640 1 --2- 172.21.15.62:0/679606076 >> v2:172.21.15.71:3300/0 conn(0x7fbf5813cea0 0x7fbf5813f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:29.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf5da5b640 1 -- 172.21.15.62:0/679606076 >> v1:172.21.15.71:6789/0 conn(0x7fbf580a2ef0 legacy=0x7fbf581379a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:29.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf5da5b640 10 monclient: _finish_hunting 0 2024-04-19T01:42:29.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf5da5b640 1 monclient: found mon.noname-b 2024-04-19T01:42:29.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf5da5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:29.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf5da5b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:42:29.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf5da5b640 1 -- 172.21.15.62:0/679606076 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf5809c350 con 0x7fbf5813a570 2024-04-19T01:42:29.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf5ea5d640 10 monclient: get_auth_request con 0x7fbf5813cea0 auth_method 0 2024-04-19T01:42:29.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.606+0000 7fbf5ea5d640 1 --2- 172.21.15.62:0/679606076 >> v2:172.21.15.71:3300/0 conn(0x7fbf5813cea0 0x7fbf5813f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:42:29.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5da5b640 1 -- 172.21.15.62:0/679606076 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbf5813a570 msgr2=0x7fbf5813c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:29.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5da5b640 1 --2- 172.21.15.62:0/679606076 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbf5813a570 0x7fbf5813c960 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbf54009870 tx=0x7fbf5401c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f64930b6a5194704 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:29.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5d25a640 1 -- 172.21.15.62:0/679606076 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf540b0070 con 0x7fbf5813a570 2024-04-19T01:42:29.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5d25a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:29.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5d25a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:42:29.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5d25a640 10 monclient: dump: 2024-04-19T01:42:29.611 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:29.611 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:29.611 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:29.611 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:29.611 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:29.611 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:29.611 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:29.611 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:29.612 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:29.612 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:29.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5d25a640 10 monclient: _finish_auth 0 2024-04-19T01:42:29.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5d25a640 10 monclient: _check_auth_tickets 2024-04-19T01:42:29.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5d25a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:29.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5d25a640 1 -- 172.21.15.62:0/679606076 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf540ab600 con 0x7fbf5813a570 2024-04-19T01:42:29.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 5 monclient: authenticate success, global_id 4325 2024-04-19T01:42:29.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:42:29.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5d25a640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:29.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:29.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5d25a640 1 -- 172.21.15.62:0/679606076 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf540ab9d0 con 0x7fbf5813a570 2024-04-19T01:42:29.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 4 set_mon_vals no callback set 2024-04-19T01:42:29.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5d25a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:29.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5d25a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:29.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5d25a640 10 monclient: dump: 2024-04-19T01:42:29.613 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:29.613 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:29.613 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:29.613 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:29.613 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:29.613 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:29.613 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:29.613 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:29.613 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:29.613 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:29.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 10 monclient: shutdown 2024-04-19T01:42:29.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:29.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 1 -- 172.21.15.62:0/679606076 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbf5813a570 msgr2=0x7fbf5813c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:29.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 1 --2- 172.21.15.62:0/679606076 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbf5813a570 0x7fbf5813c960 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbf54009870 tx=0x7fbf5401c3c0 comp rx=0 tx=0).stop 2024-04-19T01:42:29.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 1 -- 172.21.15.62:0/679606076 shutdown_connections 2024-04-19T01:42:29.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 1 --2- 172.21.15.62:0/679606076 >> v2:172.21.15.71:3300/0 conn(0x7fbf5813cea0 0x7fbf5813f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:29.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 1 --2- 172.21.15.62:0/679606076 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbf5813a570 0x7fbf5813c960 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:29.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 1 -- 172.21.15.62:0/679606076 >> 172.21.15.62:0/679606076 conn(0x7fbf580a23d0 msgr2=0x7fbf580a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:29.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 1 -- 172.21.15.62:0/679606076 shutdown_connections 2024-04-19T01:42:29.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 1 -- 172.21.15.62:0/679606076 wait complete. 2024-04-19T01:42:29.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 10 monclient: build_initial_monmap 2024-04-19T01:42:29.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 10 monclient: monmap: 2024-04-19T01:42:29.619 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:29.619 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:29.619 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:29.619 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:29.619 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:29.619 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:29.619 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:29.620 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:29.620 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:29.620 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:29.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 1 Processor -- start 2024-04-19T01:42:29.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 1 -- start start 2024-04-19T01:42:29.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 10 monclient: init 2024-04-19T01:42:29.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:29.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-19T01:42:29.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf580a2ef0 0x7fbf581cf310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:29.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 10 monclient(hunting): picked mon.noname-b con 0x7fbf580a2ef0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:29.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbf5813a570 0x7fbf581361d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:29.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 10 monclient(hunting): picked mon.noname-c con 0x7fbf5813a570 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:29.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf5813cea0 0x7fbf5811cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:29.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 10 monclient(hunting): picked mon.noname-a con 0x7fbf5813cea0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:29.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:29.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbf58141ff0 con 0x7fbf5813cea0 2024-04-19T01:42:29.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:29.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fbf58072520 con 0x7fbf580a2ef0 2024-04-19T01:42:29.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:29.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fbf58142170 con 0x7fbf5813a570 2024-04-19T01:42:29.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:29.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf604c0640 10 monclient(hunting): authenticate will time out at 935.435424s 2024-04-19T01:42:29.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5da5b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbf5813a570 0x7fbf581361d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:29.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5da5b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbf5813a570 0x7fbf581361d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60584/0 (socket says 172.21.15.62:60584) 2024-04-19T01:42:29.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5da5b640 1 -- 172.21.15.62:0/3147414349 learned_addr learned my addr 172.21.15.62:0/3147414349 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:29.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5da5b640 10 monclient(hunting): get_auth_request con 0x7fbf5813a570 auth_method 0 2024-04-19T01:42:29.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5da5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:29.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5da5b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:29.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5da5b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:29.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5ea5d640 1 --2- 172.21.15.62:0/3147414349 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf5813cea0 0x7fbf5811cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:29.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5ea5d640 10 monclient(hunting): get_auth_request con 0x7fbf5813cea0 auth_method 0 2024-04-19T01:42:29.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5ea5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:29.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5ea5d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:29.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5ea5d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:29.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5da5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:29.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5da5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:29.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5da5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:29.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5ea5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:29.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5ea5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:29.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5e25c640 1 --2- 172.21.15.62:0/3147414349 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf580a2ef0 0x7fbf581cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:29.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5ea5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:29.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5e25c640 10 monclient(hunting): get_auth_request con 0x7fbf580a2ef0 auth_method 0 2024-04-19T01:42:29.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5e25c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:29.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5e25c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:29.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5e25c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:29.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5da5b640 10 monclient(hunting): handle_auth_done global_id 4328 payload 824 2024-04-19T01:42:29.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5da5b640 1 -- 172.21.15.62:0/3147414349 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf580a2ef0 msgr2=0x7fbf581cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:29.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5da5b640 1 --2- 172.21.15.62:0/3147414349 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf580a2ef0 0x7fbf581cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:29.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5da5b640 1 -- 172.21.15.62:0/3147414349 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf5813cea0 msgr2=0x7fbf5811cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:29.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5da5b640 1 --2- 172.21.15.62:0/3147414349 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf5813cea0 0x7fbf5811cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:29.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5da5b640 10 monclient: _finish_hunting 0 2024-04-19T01:42:29.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5da5b640 1 monclient: found mon.noname-c 2024-04-19T01:42:29.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5da5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:29.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5da5b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:42:29.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5da5b640 1 -- 172.21.15.62:0/3147414349 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf58120b80 con 0x7fbf5813a570 2024-04-19T01:42:29.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5e25c640 1 --2- 172.21.15.62:0/3147414349 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf580a2ef0 0x7fbf581cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T01:42:29.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.610+0000 7fbf5ea5d640 1 --2- 172.21.15.62:0/3147414349 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf5813cea0 0x7fbf5811cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:42:29.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.614+0000 7fbf5da5b640 1 -- 172.21.15.62:0/3147414349 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbf5813a570 msgr2=0x7fbf581361d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:29.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.614+0000 7fbf5da5b640 1 --2- 172.21.15.62:0/3147414349 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbf5813a570 0x7fbf581361d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbf540ae040 tx=0x7fbf540191c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:29.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.614+0000 7fbf4effd640 1 -- 172.21.15.62:0/3147414349 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf540b0040 con 0x7fbf5813a570 2024-04-19T01:42:29.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.614+0000 7fbf4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:29.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.614+0000 7fbf4effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:29.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.614+0000 7fbf4effd640 10 monclient: dump: 2024-04-19T01:42:29.624 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:29.624 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:29.624 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:29.624 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:29.624 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:29.624 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:29.624 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:29.624 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:29.625 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:29.625 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:29.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.614+0000 7fbf4effd640 10 monclient: _finish_auth 0 2024-04-19T01:42:29.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.614+0000 7fbf4effd640 10 monclient: _check_auth_tickets 2024-04-19T01:42:29.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.614+0000 7fbf4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:29.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.614+0000 7fbf4effd640 1 -- 172.21.15.62:0/3147414349 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf540b9070 con 0x7fbf5813a570 2024-04-19T01:42:29.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.614+0000 7fbf604c0640 5 monclient: authenticate success, global_id 4328 2024-04-19T01:42:29.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.614+0000 7fbf604c0640 10 monclient: _renew_subs 2024-04-19T01:42:29.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.614+0000 7fbf604c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:29.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.614+0000 7fbf604c0640 1 -- 172.21.15.62:0/3147414349 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf581d0200 con 0x7fbf5813a570 2024-04-19T01:42:29.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.614+0000 7fbf4effd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:29.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.614+0000 7fbf4effd640 1 -- 172.21.15.62:0/3147414349 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf540b37f0 con 0x7fbf5813a570 2024-04-19T01:42:29.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.614+0000 7fbf4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:29.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.614+0000 7fbf4effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:29.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.614+0000 7fbf4effd640 10 monclient: dump: 2024-04-19T01:42:29.626 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:29.626 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:29.626 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:29.626 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:29.626 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:29.626 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:29.626 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:29.626 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:29.626 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:29.626 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:29.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.614+0000 7fbf5ca59640 4 set_mon_vals no callback set 2024-04-19T01:42:29.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.614+0000 7fbf604c0640 10 monclient: _renew_subs 2024-04-19T01:42:29.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.614+0000 7fbf604c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:29.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.614+0000 7fbf604c0640 1 -- 172.21.15.62:0/3147414349 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf581208a0 con 0x7fbf5813a570 2024-04-19T01:42:29.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.614+0000 7fbf604c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:29.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.614+0000 7fbf604c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:29.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.614+0000 7fbf604c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:29.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.614+0000 7fbf604c0640 1 -- 172.21.15.62:0/3147414349 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf580692b0 con 0x7fbf5813a570 2024-04-19T01:42:29.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.614+0000 7fbf4effd640 1 -- 172.21.15.62:0/3147414349 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fbf540b39d0 con 0x7fbf5813a570 2024-04-19T01:42:29.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.614+0000 7fbf4effd640 1 --2- 172.21.15.62:0/3147414349 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fbf3c036420 0x7fbf3c0388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:29.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.614+0000 7fbf4effd640 1 -- 172.21.15.62:0/3147414349 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4346+0+0 (secure 0 0 0) 0x7fbf540e3240 con 0x7fbf5813a570 2024-04-19T01:42:29.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.614+0000 7fbf5e25c640 1 --2- 172.21.15.62:0/3147414349 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fbf3c036420 0x7fbf3c0388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:29.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.618+0000 7fbf5e25c640 10 monclient: get_auth_request con 0x7fbf3c036420 auth_method 0 2024-04-19T01:42:29.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.618+0000 7fbf5e25c640 1 --2- 172.21.15.62:0/3147414349 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fbf3c036420 0x7fbf3c0388e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbf48009890 tx=0x7fbf48008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:29.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.618+0000 7fbf4effd640 1 -- 172.21.15.62:0/3147414349 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fbf54016130 con 0x7fbf5813a570 2024-04-19T01:42:29.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.618+0000 7fbf4effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:29.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.618+0000 7fbf4effd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:29.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.774+0000 7fbf604c0640 1 -- 172.21.15.62:0/3147414349 --> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbf5806b980 con 0x7fbf3c036420 2024-04-19T01:42:29.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.778+0000 7fbf4effd640 1 -- 172.21.15.62:0/3147414349 <== mgr.4108 v2:172.21.15.71:6824/2133533992 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33750 (secure 0 0 0) 0x7fbf5806b980 con 0x7fbf3c036420 2024-04-19T01:42:29.780 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:42:29.780 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-04-19T01:42:29.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.778+0000 7fbf604c0640 1 -- 172.21.15.62:0/3147414349 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fbf3c036420 msgr2=0x7fbf3c0388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:29.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.778+0000 7fbf604c0640 1 --2- 172.21.15.62:0/3147414349 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fbf3c036420 0x7fbf3c0388e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbf48009890 tx=0x7fbf48008040 comp rx=0 tx=0).stop 2024-04-19T01:42:29.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.778+0000 7fbf604c0640 10 monclient: shutdown 2024-04-19T01:42:29.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.778+0000 7fbf604c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:29.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.778+0000 7fbf604c0640 1 -- 172.21.15.62:0/3147414349 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbf5813a570 msgr2=0x7fbf581361d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:29.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.778+0000 7fbf604c0640 1 --2- 172.21.15.62:0/3147414349 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbf5813a570 0x7fbf581361d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbf540ae040 tx=0x7fbf540191c0 comp rx=0 tx=0).stop 2024-04-19T01:42:29.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.782+0000 7fbf604c0640 1 -- 172.21.15.62:0/3147414349 shutdown_connections 2024-04-19T01:42:29.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.782+0000 7fbf604c0640 1 --2- 172.21.15.62:0/3147414349 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fbf3c036420 0x7fbf3c0388e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:29.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.782+0000 7fbf604c0640 1 --2- 172.21.15.62:0/3147414349 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbf5813a570 0x7fbf581361d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:29.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.782+0000 7fbf604c0640 1 --2- 172.21.15.62:0/3147414349 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf5813cea0 0x7fbf5811cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:29.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.782+0000 7fbf604c0640 1 --2- 172.21.15.62:0/3147414349 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf580a2ef0 0x7fbf581cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:29.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.782+0000 7fbf604c0640 1 -- 172.21.15.62:0/3147414349 >> 172.21.15.62:0/3147414349 conn(0x7fbf580a23d0 msgr2=0x7fbf580a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:29.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.782+0000 7fbf604c0640 1 -- 172.21.15.62:0/3147414349 shutdown_connections 2024-04-19T01:42:29.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.782+0000 7fbf604c0640 1 -- 172.21.15.62:0/3147414349 wait complete. 2024-04-19T01:42:29.797 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-04-19T01:42:28.705328+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":101,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164112,"kb_used_data":1856,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066928,"statfs":{"total":579820584960,"available":579652534272,"internally_reserved":0,"allocated":1900544,"data_stored":1317428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.681885"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.163776+0000","last_change":"2024-04-19T01:42:24.162228+0000","last_active":"2024-04-19T01:42:24.163776+0000","last_peered":"2024-04-19T01:42:24.163776+0000","last_clean":"2024-04-19T01:42:24.163776+0000","last_became_active":"2024-04-19T01:42:24.161645+0000","last_became_peered":"2024-04-19T01:42:24.161645+0000","last_unstale":"2024-04-19T01:42:24.163776+0000","last_undegraded":"2024-04-19T01:42:24.163776+0000","last_fullsized":"2024-04-19T01:42:24.163776+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:08:44.489901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.141485+0000","last_change":"2024-04-19T01:42:24.132804+0000","last_active":"2024-04-19T01:42:24.141485+0000","last_peered":"2024-04-19T01:42:24.141485+0000","last_clean":"2024-04-19T01:42:24.141485+0000","last_became_active":"2024-04-19T01:42:24.131937+0000","last_became_peered":"2024-04-19T01:42:24.131937+0000","last_unstale":"2024-04-19T01:42:24.141485+0000","last_undegraded":"2024-04-19T01:42:24.141485+0000","last_fullsized":"2024-04-19T01:42:24.141485+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T06:51:05.320754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.139253+0000","last_change":"2024-04-19T01:42:24.135590+0000","last_active":"2024-04-19T01:42:24.139253+0000","last_peered":"2024-04-19T01:42:24.139253+0000","last_clean":"2024-04-19T01:42:24.139253+0000","last_became_active":"2024-04-19T01:42:24.134918+0000","last_became_peered":"2024-04-19T01:42:24.134918+0000","last_unstale":"2024-04-19T01:42:24.139253+0000","last_undegraded":"2024-04-19T01:42:24.139253+0000","last_fullsized":"2024-04-19T01:42:24.139253+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T08:09:26.534746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.141458+0000","last_change":"2024-04-19T01:42:24.132575+0000","last_active":"2024-04-19T01:42:24.141458+0000","last_peered":"2024-04-19T01:42:24.141458+0000","last_clean":"2024-04-19T01:42:24.141458+0000","last_became_active":"2024-04-19T01:42:24.131894+0000","last_became_peered":"2024-04-19T01:42:24.131894+0000","last_unstale":"2024-04-19T01:42:24.141458+0000","last_undegraded":"2024-04-19T01:42:24.141458+0000","last_fullsized":"2024-04-19T01:42:24.141458+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:25:51.912215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.148595+0000","last_change":"2024-04-19T01:42:24.145699+0000","last_active":"2024-04-19T01:42:24.148595+0000","last_peered":"2024-04-19T01:42:24.148595+0000","last_clean":"2024-04-19T01:42:24.148595+0000","last_became_active":"2024-04-19T01:42:24.144746+0000","last_became_peered":"2024-04-19T01:42:24.144746+0000","last_unstale":"2024-04-19T01:42:24.148595+0000","last_undegraded":"2024-04-19T01:42:24.148595+0000","last_fullsized":"2024-04-19T01:42:24.148595+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T04:20:46.381920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.145097+0000","last_change":"2024-04-19T01:42:24.142754+0000","last_active":"2024-04-19T01:42:24.145097+0000","last_peered":"2024-04-19T01:42:24.145097+0000","last_clean":"2024-04-19T01:42:24.145097+0000","last_became_active":"2024-04-19T01:42:24.142309+0000","last_became_peered":"2024-04-19T01:42:24.142309+0000","last_unstale":"2024-04-19T01:42:24.145097+0000","last_undegraded":"2024-04-19T01:42:24.145097+0000","last_fullsized":"2024-04-19T01:42:24.145097+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:18:58.995723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.136907+0000","last_change":"2024-04-19T01:42:24.134484+0000","last_active":"2024-04-19T01:42:24.136907+0000","last_peered":"2024-04-19T01:42:24.136907+0000","last_clean":"2024-04-19T01:42:24.136907+0000","last_became_active":"2024-04-19T01:42:24.133282+0000","last_became_peered":"2024-04-19T01:42:24.133282+0000","last_unstale":"2024-04-19T01:42:24.136907+0000","last_undegraded":"2024-04-19T01:42:24.136907+0000","last_fullsized":"2024-04-19T01:42:24.136907+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T06:02:03.349002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.149236+0000","last_change":"2024-04-19T01:42:24.146550+0000","last_active":"2024-04-19T01:42:24.149236+0000","last_peered":"2024-04-19T01:42:24.149236+0000","last_clean":"2024-04-19T01:42:24.149236+0000","last_became_active":"2024-04-19T01:42:24.145667+0000","last_became_peered":"2024-04-19T01:42:24.145667+0000","last_unstale":"2024-04-19T01:42:24.149236+0000","last_undegraded":"2024-04-19T01:42:24.149236+0000","last_fullsized":"2024-04-19T01:42:24.149236+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:48:51.752906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":165,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.130442+0000","last_change":"2024-04-19T01:42:22.132251+0000","last_active":"2024-04-19T01:42:24.130442+0000","last_peered":"2024-04-19T01:42:24.130442+0000","last_clean":"2024-04-19T01:42:24.130442+0000","last_became_active":"2024-04-19T01:42:22.131432+0000","last_became_peered":"2024-04-19T01:42:22.131432+0000","last_unstale":"2024-04-19T01:42:24.130442+0000","last_undegraded":"2024-04-19T01:42:24.130442+0000","last_fullsized":"2024-04-19T01:42:24.130442+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:21.102542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:21.102542+0000","last_clean_scrub_stamp":"2024-04-19T01:42:21.102542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T06:47:12.046147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":101,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":101,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607555,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":139264,"data_stored":43262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":139264,"data_stored":43262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344164,"statfs":{"total":96636764160,"available":96608423936,"internally_reserved":0,"allocated":671744,"data_stored":572190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":139264,"data_stored":43262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":139264,"data_stored":43262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":671744,"data_stored":572190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:42:29.798 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:42:29.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.906+0000 7ff1b9bad640 10 monclient: get_monmap_and_config 2024-04-19T01:42:29.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.906+0000 7ff1b9bad640 10 monclient: build_initial_monmap 2024-04-19T01:42:29.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.906+0000 7ff1b9bad640 10 monclient: monmap: 2024-04-19T01:42:29.909 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:29.909 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:29.909 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:29.913180+0000 2024-04-19T01:42:29.909 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:29.913180+0000 2024-04-19T01:42:29.909 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:29.909 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:29.909 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:29.909 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:29.909 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:29.909 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:29.910 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:29.910 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:29.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.906+0000 7ff1b9bad640 1 Processor -- start 2024-04-19T01:42:29.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.906+0000 7ff1b9bad640 1 -- start start 2024-04-19T01:42:29.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.906+0000 7ff1b9bad640 10 monclient: init 2024-04-19T01:42:29.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b9bad640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:29.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b9bad640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-04-19T01:42:29.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b9bad640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff1b40a47e0 addr v1:172.21.15.71:6789/0 2024-04-19T01:42:29.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b9bad640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1b40a4bc0 0x7ff1b40a4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:29.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b9bad640 10 monclient(hunting): picked mon.noname-b con 0x7ff1b40a4bc0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:29.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b9bad640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff1b4139560 0x7ff1b413b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:29.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b9bad640 10 monclient(hunting): picked mon.noname-c con 0x7ff1b4139560 addr v2:172.21.15.71:3300/0 2024-04-19T01:42:29.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b9bad640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1b409fa60 con 0x7ff1b40a47e0 2024-04-19T01:42:29.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b9bad640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:29.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b9bad640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7ff1b409e690 con 0x7ff1b4139560 2024-04-19T01:42:29.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b9bad640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:29.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b9bad640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7ff1b409ec50 con 0x7ff1b40a4bc0 2024-04-19T01:42:29.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b9bad640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:29.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b9bad640 10 monclient(hunting): authenticate will time out at 935.732630s 2024-04-19T01:42:29.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b2ffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1b40a4bc0 0x7ff1b40a4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:29.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b2ffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1b40a4bc0 0x7ff1b40a4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60600/0 (socket says 172.21.15.62:60600) 2024-04-19T01:42:29.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b2ffd640 1 -- 172.21.15.62:0/2070580084 learned_addr learned my addr 172.21.15.62:0/2070580084 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:29.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b2ffd640 10 monclient(hunting): get_auth_request con 0x7ff1b40a4bc0 auth_method 0 2024-04-19T01:42:29.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:29.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:29.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:29.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b3fff640 1 --2- 172.21.15.62:0/2070580084 >> v2:172.21.15.71:3300/0 conn(0x7ff1b4139560 0x7ff1b413b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:29.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:29.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:29.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:29.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b3fff640 10 monclient(hunting): get_auth_request con 0x7ff1b4139560 auth_method 0 2024-04-19T01:42:29.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:29.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:29.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:29.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:29.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:29.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:29.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b2ffd640 10 monclient(hunting): handle_auth_done global_id 4331 payload 274 2024-04-19T01:42:29.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b2ffd640 1 -- 172.21.15.62:0/2070580084 >> v2:172.21.15.71:3300/0 conn(0x7ff1b4139560 msgr2=0x7ff1b413b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:29.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b2ffd640 1 --2- 172.21.15.62:0/2070580084 >> v2:172.21.15.71:3300/0 conn(0x7ff1b4139560 0x7ff1b413b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:29.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b2ffd640 1 -- 172.21.15.62:0/2070580084 >> v1:172.21.15.71:6789/0 conn(0x7ff1b40a47e0 legacy=0x7ff1b409f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:29.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b2ffd640 10 monclient: _finish_hunting 0 2024-04-19T01:42:29.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b2ffd640 1 monclient: found mon.noname-b 2024-04-19T01:42:29.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:29.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:42:29.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.910+0000 7ff1b2ffd640 1 -- 172.21.15.62:0/2070580084 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1b409e970 con 0x7ff1b40a4bc0 2024-04-19T01:42:29.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b2ffd640 1 -- 172.21.15.62:0/2070580084 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1b40a4bc0 msgr2=0x7ff1b40a4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:29.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b2ffd640 1 --2- 172.21.15.62:0/2070580084 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1b40a4bc0 0x7ff1b40a4fe0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff19c009870 tx=0x7ff19c01c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86ae7da138bc093b server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:29.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b27fc640 1 -- 172.21.15.62:0/2070580084 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff19c0b2070 con 0x7ff1b40a4bc0 2024-04-19T01:42:29.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:29.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:42:29.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b27fc640 10 monclient: dump: 2024-04-19T01:42:29.919 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:29.919 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:29.919 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:29.919 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:29.919 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:29.919 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:29.919 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:29.919 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:29.919 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:29.920 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:29.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b27fc640 10 monclient: _finish_auth 0 2024-04-19T01:42:29.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b27fc640 10 monclient: _check_auth_tickets 2024-04-19T01:42:29.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:29.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b27fc640 1 -- 172.21.15.62:0/2070580084 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff19c0ad640 con 0x7ff1b40a4bc0 2024-04-19T01:42:29.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b9bad640 5 monclient: authenticate success, global_id 4331 2024-04-19T01:42:29.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b9bad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:42:29.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b27fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:29.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b9bad640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:29.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b27fc640 1 -- 172.21.15.62:0/2070580084 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff19c0ada10 con 0x7ff1b40a4bc0 2024-04-19T01:42:29.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:29.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b9bad640 4 set_mon_vals no callback set 2024-04-19T01:42:29.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:29.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b27fc640 10 monclient: dump: 2024-04-19T01:42:29.921 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:29.921 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:29.921 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:29.921 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:29.921 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:29.921 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:29.921 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:29.921 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:29.921 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:29.921 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:29.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b9bad640 10 monclient: shutdown 2024-04-19T01:42:29.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b9bad640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:29.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b9bad640 1 -- 172.21.15.62:0/2070580084 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1b40a4bc0 msgr2=0x7ff1b40a4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:29.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b9bad640 1 --2- 172.21.15.62:0/2070580084 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1b40a4bc0 0x7ff1b40a4fe0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff19c009870 tx=0x7ff19c01c3f0 comp rx=0 tx=0).stop 2024-04-19T01:42:29.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b9bad640 1 -- 172.21.15.62:0/2070580084 shutdown_connections 2024-04-19T01:42:29.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b9bad640 1 --2- 172.21.15.62:0/2070580084 >> v2:172.21.15.71:3300/0 conn(0x7ff1b4139560 0x7ff1b413b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:29.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b9bad640 1 --2- 172.21.15.62:0/2070580084 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1b40a4bc0 0x7ff1b40a4fe0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:29.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b9bad640 1 -- 172.21.15.62:0/2070580084 >> 172.21.15.62:0/2070580084 conn(0x7ff1b40a23d0 msgr2=0x7ff1b40a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:29.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b9bad640 1 -- 172.21.15.62:0/2070580084 shutdown_connections 2024-04-19T01:42:29.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b9bad640 1 -- 172.21.15.62:0/2070580084 wait complete. 2024-04-19T01:42:29.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b9bad640 10 monclient: build_initial_monmap 2024-04-19T01:42:29.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b9bad640 10 monclient: monmap: 2024-04-19T01:42:29.922 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:29.922 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:29.922 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:29.922 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:29.922 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:29.922 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:29.923 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:29.923 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:29.923 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:29.923 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:29.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b9bad640 1 Processor -- start 2024-04-19T01:42:29.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b9bad640 1 -- start start 2024-04-19T01:42:29.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.914+0000 7ff1b9bad640 10 monclient: init 2024-04-19T01:42:29.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b9bad640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:29.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b9bad640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-19T01:42:29.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b9bad640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1b40a47e0 0x7ff1b41cb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:29.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b9bad640 10 monclient(hunting): picked mon.noname-c con 0x7ff1b40a47e0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:29.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b9bad640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1b40a4bc0 0x7ff1b41cb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:29.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b9bad640 10 monclient(hunting): picked mon.noname-a con 0x7ff1b40a4bc0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:29.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b9bad640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1b4139560 0x7ff1b41cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:29.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b9bad640 10 monclient(hunting): picked mon.noname-b con 0x7ff1b4139560 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:29.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b9bad640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:29.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b9bad640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b409be30 con 0x7ff1b40a4bc0 2024-04-19T01:42:29.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b9bad640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:29.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b9bad640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b409bcb0 con 0x7ff1b4139560 2024-04-19T01:42:29.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b9bad640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:29.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b9bad640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7ff1b409bfb0 con 0x7ff1b40a47e0 2024-04-19T01:42:29.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b9bad640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:29.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b9bad640 10 monclient(hunting): authenticate will time out at 935.740125s 2024-04-19T01:42:29.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b37fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1b40a47e0 0x7ff1b41cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:29.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b37fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1b40a47e0 0x7ff1b41cb490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60610/0 (socket says 172.21.15.62:60610) 2024-04-19T01:42:29.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b37fe640 1 -- 172.21.15.62:0/2603183357 learned_addr learned my addr 172.21.15.62:0/2603183357 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:29.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b37fe640 10 monclient(hunting): get_auth_request con 0x7ff1b40a47e0 auth_method 0 2024-04-19T01:42:29.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:29.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:29.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:29.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b2ffd640 1 --2- 172.21.15.62:0/2603183357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1b40a4bc0 0x7ff1b41cb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:29.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b2ffd640 10 monclient(hunting): get_auth_request con 0x7ff1b40a4bc0 auth_method 0 2024-04-19T01:42:29.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:29.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:29.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:29.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:29.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:29.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:29.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:29.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:29.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:29.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b3fff640 1 --2- 172.21.15.62:0/2603183357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1b4139560 0x7ff1b41cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:29.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b3fff640 10 monclient(hunting): get_auth_request con 0x7ff1b4139560 auth_method 0 2024-04-19T01:42:29.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:29.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:29.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:29.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b37fe640 10 monclient(hunting): handle_auth_done global_id 4334 payload 824 2024-04-19T01:42:29.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b37fe640 1 -- 172.21.15.62:0/2603183357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1b4139560 msgr2=0x7ff1b41cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:29.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b37fe640 1 --2- 172.21.15.62:0/2603183357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1b4139560 0x7ff1b41cbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:29.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b37fe640 1 -- 172.21.15.62:0/2603183357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1b40a4bc0 msgr2=0x7ff1b41cb9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:29.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b37fe640 1 --2- 172.21.15.62:0/2603183357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1b40a4bc0 0x7ff1b41cb9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:29.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b37fe640 10 monclient: _finish_hunting 0 2024-04-19T01:42:29.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b37fe640 1 monclient: found mon.noname-c 2024-04-19T01:42:29.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:29.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:42:29.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b37fe640 1 -- 172.21.15.62:0/2603183357 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1b4072820 con 0x7ff1b40a47e0 2024-04-19T01:42:29.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b2ffd640 1 --2- 172.21.15.62:0/2603183357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1b40a4bc0 0x7ff1b41cb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:42:29.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b37fe640 1 -- 172.21.15.62:0/2603183357 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1b40a47e0 msgr2=0x7ff1b41cb490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:29.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b37fe640 1 --2- 172.21.15.62:0/2603183357 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1b40a47e0 0x7ff1b41cb490 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac00c7c0 tx=0x7ff1ac00cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:29.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1abfff640 1 -- 172.21.15.62:0/2603183357 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac016020 con 0x7ff1b40a47e0 2024-04-19T01:42:29.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:29.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1abfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:29.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1abfff640 10 monclient: dump: 2024-04-19T01:42:29.929 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:29.929 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:29.929 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:29.929 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:29.929 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:29.929 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:29.929 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:29.929 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:29.929 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:29.929 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:29.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1abfff640 10 monclient: _finish_auth 0 2024-04-19T01:42:29.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1abfff640 10 monclient: _check_auth_tickets 2024-04-19T01:42:29.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:29.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b9bad640 5 monclient: authenticate success, global_id 4334 2024-04-19T01:42:29.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1abfff640 1 -- 172.21.15.62:0/2603183357 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1ac00ac60 con 0x7ff1b40a47e0 2024-04-19T01:42:29.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b9bad640 10 monclient: _renew_subs 2024-04-19T01:42:29.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b9bad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:29.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b9bad640 1 -- 172.21.15.62:0/2603183357 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1b409e000 con 0x7ff1b40a47e0 2024-04-19T01:42:29.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1abfff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:29.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1abfff640 1 -- 172.21.15.62:0/2603183357 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac012660 con 0x7ff1b40a47e0 2024-04-19T01:42:29.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:29.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1abfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:29.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1abfff640 10 monclient: dump: 2024-04-19T01:42:29.930 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:29.930 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:29.931 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:29.931 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:29.931 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:29.931 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:29.931 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:29.931 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:29.931 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:29.931 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:29.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b1ffb640 4 set_mon_vals no callback set 2024-04-19T01:42:29.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b9bad640 10 monclient: _renew_subs 2024-04-19T01:42:29.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b9bad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:29.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b9bad640 1 -- 172.21.15.62:0/2603183357 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1b411d1d0 con 0x7ff1b40a47e0 2024-04-19T01:42:29.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b9bad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:29.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b9bad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:29.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b9bad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:29.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.918+0000 7ff1b9bad640 1 -- 172.21.15.62:0/2603183357 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1b40692b0 con 0x7ff1b40a47e0 2024-04-19T01:42:29.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.922+0000 7ff1abfff640 1 -- 172.21.15.62:0/2603183357 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7ff1ac019020 con 0x7ff1b40a47e0 2024-04-19T01:42:29.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.922+0000 7ff1abfff640 1 --2- 172.21.15.62:0/2603183357 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7ff194036470 0x7ff194038930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:29.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.922+0000 7ff1abfff640 1 -- 172.21.15.62:0/2603183357 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4346+0+0 (secure 0 0 0) 0x7ff1ac0497d0 con 0x7ff1b40a47e0 2024-04-19T01:42:29.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.922+0000 7ff1b2ffd640 1 --2- 172.21.15.62:0/2603183357 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7ff194036470 0x7ff194038930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:29.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.922+0000 7ff1b2ffd640 10 monclient: get_auth_request con 0x7ff194036470 auth_method 0 2024-04-19T01:42:29.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.926+0000 7ff1b2ffd640 1 --2- 172.21.15.62:0/2603183357 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7ff194036470 0x7ff194038930 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff19c0ac000 tx=0x7ff19c0a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:29.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.926+0000 7ff1abfff640 1 -- 172.21.15.62:0/2603183357 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff1ac020030 con 0x7ff1b40a47e0 2024-04-19T01:42:29.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.926+0000 7ff1abfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:29.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:29.926+0000 7ff1abfff640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:30.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.082+0000 7ff1b9bad640 1 -- 172.21.15.62:0/2603183357 --> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff1b40008e0 con 0x7ff194036470 2024-04-19T01:42:30.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.086+0000 7ff1abfff640 1 -- 172.21.15.62:0/2603183357 <== mgr.4108 v2:172.21.15.71:6824/2133533992 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33750 (secure 0 0 0) 0x7ff1b40008e0 con 0x7ff194036470 2024-04-19T01:42:30.087 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:42:30.087 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-04-19T01:42:30.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.086+0000 7ff1b9bad640 1 -- 172.21.15.62:0/2603183357 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7ff194036470 msgr2=0x7ff194038930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:30.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.086+0000 7ff1b9bad640 1 --2- 172.21.15.62:0/2603183357 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7ff194036470 0x7ff194038930 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff19c0ac000 tx=0x7ff19c0a7070 comp rx=0 tx=0).stop 2024-04-19T01:42:30.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.086+0000 7ff1b9bad640 10 monclient: shutdown 2024-04-19T01:42:30.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.086+0000 7ff1b9bad640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:30.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.086+0000 7ff1b9bad640 1 -- 172.21.15.62:0/2603183357 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1b40a47e0 msgr2=0x7ff1b41cb490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:30.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.086+0000 7ff1b9bad640 1 --2- 172.21.15.62:0/2603183357 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1b40a47e0 0x7ff1b41cb490 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac00c7c0 tx=0x7ff1ac00cc80 comp rx=0 tx=0).stop 2024-04-19T01:42:30.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.086+0000 7ff1b9bad640 1 -- 172.21.15.62:0/2603183357 shutdown_connections 2024-04-19T01:42:30.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.086+0000 7ff1b9bad640 1 --2- 172.21.15.62:0/2603183357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1b4139560 0x7ff1b41cbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.086+0000 7ff1b9bad640 1 --2- 172.21.15.62:0/2603183357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1b40a4bc0 0x7ff1b41cb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.086+0000 7ff1b9bad640 1 --2- 172.21.15.62:0/2603183357 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7ff194036470 0x7ff194038930 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.086+0000 7ff1b9bad640 1 --2- 172.21.15.62:0/2603183357 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff1b40a47e0 0x7ff1b41cb490 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.086+0000 7ff1b9bad640 1 -- 172.21.15.62:0/2603183357 >> 172.21.15.62:0/2603183357 conn(0x7ff1b40a23d0 msgr2=0x7ff1b4138b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:30.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.086+0000 7ff1b9bad640 1 -- 172.21.15.62:0/2603183357 shutdown_connections 2024-04-19T01:42:30.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.086+0000 7ff1b9bad640 1 -- 172.21.15.62:0/2603183357 wait complete. 2024-04-19T01:42:30.104 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-04-19T01:42:28.705328+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":101,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164112,"kb_used_data":1856,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066928,"statfs":{"total":579820584960,"available":579652534272,"internally_reserved":0,"allocated":1900544,"data_stored":1317428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.681885"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.163776+0000","last_change":"2024-04-19T01:42:24.162228+0000","last_active":"2024-04-19T01:42:24.163776+0000","last_peered":"2024-04-19T01:42:24.163776+0000","last_clean":"2024-04-19T01:42:24.163776+0000","last_became_active":"2024-04-19T01:42:24.161645+0000","last_became_peered":"2024-04-19T01:42:24.161645+0000","last_unstale":"2024-04-19T01:42:24.163776+0000","last_undegraded":"2024-04-19T01:42:24.163776+0000","last_fullsized":"2024-04-19T01:42:24.163776+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:08:44.489901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.141485+0000","last_change":"2024-04-19T01:42:24.132804+0000","last_active":"2024-04-19T01:42:24.141485+0000","last_peered":"2024-04-19T01:42:24.141485+0000","last_clean":"2024-04-19T01:42:24.141485+0000","last_became_active":"2024-04-19T01:42:24.131937+0000","last_became_peered":"2024-04-19T01:42:24.131937+0000","last_unstale":"2024-04-19T01:42:24.141485+0000","last_undegraded":"2024-04-19T01:42:24.141485+0000","last_fullsized":"2024-04-19T01:42:24.141485+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T06:51:05.320754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.139253+0000","last_change":"2024-04-19T01:42:24.135590+0000","last_active":"2024-04-19T01:42:24.139253+0000","last_peered":"2024-04-19T01:42:24.139253+0000","last_clean":"2024-04-19T01:42:24.139253+0000","last_became_active":"2024-04-19T01:42:24.134918+0000","last_became_peered":"2024-04-19T01:42:24.134918+0000","last_unstale":"2024-04-19T01:42:24.139253+0000","last_undegraded":"2024-04-19T01:42:24.139253+0000","last_fullsized":"2024-04-19T01:42:24.139253+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T08:09:26.534746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.141458+0000","last_change":"2024-04-19T01:42:24.132575+0000","last_active":"2024-04-19T01:42:24.141458+0000","last_peered":"2024-04-19T01:42:24.141458+0000","last_clean":"2024-04-19T01:42:24.141458+0000","last_became_active":"2024-04-19T01:42:24.131894+0000","last_became_peered":"2024-04-19T01:42:24.131894+0000","last_unstale":"2024-04-19T01:42:24.141458+0000","last_undegraded":"2024-04-19T01:42:24.141458+0000","last_fullsized":"2024-04-19T01:42:24.141458+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:25:51.912215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.148595+0000","last_change":"2024-04-19T01:42:24.145699+0000","last_active":"2024-04-19T01:42:24.148595+0000","last_peered":"2024-04-19T01:42:24.148595+0000","last_clean":"2024-04-19T01:42:24.148595+0000","last_became_active":"2024-04-19T01:42:24.144746+0000","last_became_peered":"2024-04-19T01:42:24.144746+0000","last_unstale":"2024-04-19T01:42:24.148595+0000","last_undegraded":"2024-04-19T01:42:24.148595+0000","last_fullsized":"2024-04-19T01:42:24.148595+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T04:20:46.381920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.145097+0000","last_change":"2024-04-19T01:42:24.142754+0000","last_active":"2024-04-19T01:42:24.145097+0000","last_peered":"2024-04-19T01:42:24.145097+0000","last_clean":"2024-04-19T01:42:24.145097+0000","last_became_active":"2024-04-19T01:42:24.142309+0000","last_became_peered":"2024-04-19T01:42:24.142309+0000","last_unstale":"2024-04-19T01:42:24.145097+0000","last_undegraded":"2024-04-19T01:42:24.145097+0000","last_fullsized":"2024-04-19T01:42:24.145097+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:18:58.995723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.136907+0000","last_change":"2024-04-19T01:42:24.134484+0000","last_active":"2024-04-19T01:42:24.136907+0000","last_peered":"2024-04-19T01:42:24.136907+0000","last_clean":"2024-04-19T01:42:24.136907+0000","last_became_active":"2024-04-19T01:42:24.133282+0000","last_became_peered":"2024-04-19T01:42:24.133282+0000","last_unstale":"2024-04-19T01:42:24.136907+0000","last_undegraded":"2024-04-19T01:42:24.136907+0000","last_fullsized":"2024-04-19T01:42:24.136907+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T06:02:03.349002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.149236+0000","last_change":"2024-04-19T01:42:24.146550+0000","last_active":"2024-04-19T01:42:24.149236+0000","last_peered":"2024-04-19T01:42:24.149236+0000","last_clean":"2024-04-19T01:42:24.149236+0000","last_became_active":"2024-04-19T01:42:24.145667+0000","last_became_peered":"2024-04-19T01:42:24.145667+0000","last_unstale":"2024-04-19T01:42:24.149236+0000","last_undegraded":"2024-04-19T01:42:24.149236+0000","last_fullsized":"2024-04-19T01:42:24.149236+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:48:51.752906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":165,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T01:42:24.130442+0000","last_change":"2024-04-19T01:42:22.132251+0000","last_active":"2024-04-19T01:42:24.130442+0000","last_peered":"2024-04-19T01:42:24.130442+0000","last_clean":"2024-04-19T01:42:24.130442+0000","last_became_active":"2024-04-19T01:42:22.131432+0000","last_became_peered":"2024-04-19T01:42:22.131432+0000","last_unstale":"2024-04-19T01:42:24.130442+0000","last_undegraded":"2024-04-19T01:42:24.130442+0000","last_fullsized":"2024-04-19T01:42:24.130442+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:21.102542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:21.102542+0000","last_clean_scrub_stamp":"2024-04-19T01:42:21.102542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T06:47:12.046147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":101,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":101,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607555,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":139264,"data_stored":43262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":139264,"data_stored":43262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344164,"statfs":{"total":96636764160,"available":96608423936,"internally_reserved":0,"allocated":671744,"data_stored":572190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":139264,"data_stored":43262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":139264,"data_stored":43262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":671744,"data_stored":572190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:42:30.105 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-04-19T01:42:30.105 DEBUG:tasks.mon_thrash:Yielding 2024-04-19T01:42:30.105 INFO:teuthology.run_tasks:Running task exec... 2024-04-19T01:42:30.115 INFO:teuthology.task.exec:Executing custom commands... 2024-04-19T01:42:30.115 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi071.front.sepia.ceph.com 2024-04-19T01:42:30.115 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:42:30.116 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-04-19T01:42:30.116 INFO:tasks.mon_thrash.mon_thrasher:seed: 1713490950, revive delay: 90.0, thrash delay: 1.0 thrash many: True, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-04-19T01:42:30.116 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-19T01:42:30.116 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:42:30.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f536c6f2640 10 monclient: get_monmap_and_config 2024-04-19T01:42:30.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f536c6f2640 10 monclient: build_initial_monmap 2024-04-19T01:42:30.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f536c6f2640 10 monclient: monmap: 2024-04-19T01:42:30.223 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:30.223 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:30.223 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:30.227984+0000 2024-04-19T01:42:30.224 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:30.227984+0000 2024-04-19T01:42:30.224 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:30.224 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:30.224 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:30.224 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:30.224 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:30.224 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:30.224 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:30.224 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:30.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f536c6f2640 1 Processor -- start 2024-04-19T01:42:30.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f536c6f2640 1 -- start start 2024-04-19T01:42:30.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f536c6f2640 10 monclient: init 2024-04-19T01:42:30.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f536c6f2640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:30.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f536c6f2640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-04-19T01:42:30.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f536c6f2640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f53640a47e0 0x7f53640a4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:30.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f536c6f2640 10 monclient(hunting): picked mon.noname-a con 0x7f53640a47e0 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:30.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f536c6f2640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f53640a5100 0x7f53641350b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:30.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f536c6f2640 10 monclient(hunting): picked mon.noname-c con 0x7f53640a5100 addr v2:172.21.15.71:3300/0 2024-04-19T01:42:30.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f536c6f2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f53641355f0 addr v1:172.21.15.71:6789/0 2024-04-19T01:42:30.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f536c6f2640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f536409f7c0 con 0x7f53641355f0 2024-04-19T01:42:30.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f536c6f2640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:30.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f536c6f2640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f536409ee90 con 0x7f53640a47e0 2024-04-19T01:42:30.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f536c6f2640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:30.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f536c6f2640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f536409c2d0 con 0x7f53640a5100 2024-04-19T01:42:30.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f536c6f2640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:30.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f536c6f2640 10 monclient(hunting): authenticate will time out at 936.047454s 2024-04-19T01:42:30.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f536a48e640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f53640a47e0 0x7f53640a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:30.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f536a48e640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f53640a47e0 0x7f53640a4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36496/0 (socket says 172.21.15.62:36496) 2024-04-19T01:42:30.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f536a48e640 1 -- 172.21.15.62:0/489034154 learned_addr learned my addr 172.21.15.62:0/489034154 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:30.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f536a48e640 10 monclient(hunting): get_auth_request con 0x7f53640a47e0 auth_method 0 2024-04-19T01:42:30.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f536a48e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:30.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f536a48e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:30.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f536a48e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:30.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f5369c8d640 1 --2- 172.21.15.62:0/489034154 >> v2:172.21.15.71:3300/0 conn(0x7f53640a5100 0x7f53641350b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:30.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f536a48e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:30.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f536a48e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:30.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f536a48e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:30.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f5369c8d640 10 monclient(hunting): get_auth_request con 0x7f53640a5100 auth_method 0 2024-04-19T01:42:30.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f5369c8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:30.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f5369c8d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:30.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.222+0000 7f5369c8d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:30.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.226+0000 7f5369c8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:30.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.226+0000 7f5369c8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:30.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.226+0000 7f5369c8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:30.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.226+0000 7f536a48e640 10 monclient(hunting): handle_auth_done global_id 4338 payload 274 2024-04-19T01:42:30.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.226+0000 7f536a48e640 1 -- 172.21.15.62:0/489034154 >> v2:172.21.15.71:3300/0 conn(0x7f53640a5100 msgr2=0x7f53641350b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:30.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.226+0000 7f536a48e640 1 --2- 172.21.15.62:0/489034154 >> v2:172.21.15.71:3300/0 conn(0x7f53640a5100 0x7f53641350b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.226+0000 7f536a48e640 1 -- 172.21.15.62:0/489034154 >> v1:172.21.15.71:6789/0 conn(0x7f53641355f0 legacy=0x7f536413c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:30.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.226+0000 7f536a48e640 10 monclient: _finish_hunting 0 2024-04-19T01:42:30.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.226+0000 7f536a48e640 1 monclient: found mon.noname-a 2024-04-19T01:42:30.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.226+0000 7f536a48e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:30.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.226+0000 7f536a48e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:30.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.226+0000 7f536a48e640 1 -- 172.21.15.62:0/489034154 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f536409faa0 con 0x7f53640a47e0 2024-04-19T01:42:30.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536a48e640 1 -- 172.21.15.62:0/489034154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53640a47e0 msgr2=0x7f53640a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:30.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536a48e640 1 --2- 172.21.15.62:0/489034154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53640a47e0 0x7f53640a4bc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f535c009e50 tx=0x7f535c01c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5c6e144d0c63498 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:30.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536948c640 1 -- 172.21.15.62:0/489034154 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535c0a5560 con 0x7f53640a47e0 2024-04-19T01:42:30.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536948c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:30.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536948c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:30.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536948c640 10 monclient: dump: 2024-04-19T01:42:30.234 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:30.234 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:30.234 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.234 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.234 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:30.234 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:30.235 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:30.235 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:30.235 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:30.235 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:30.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536948c640 10 monclient: _finish_auth 0 2024-04-19T01:42:30.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536948c640 10 monclient: _check_auth_tickets 2024-04-19T01:42:30.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536948c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:30.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 5 monclient: authenticate success, global_id 4338 2024-04-19T01:42:30.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:42:30.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536948c640 1 -- 172.21.15.62:0/489034154 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f535c0a5700 con 0x7f53640a47e0 2024-04-19T01:42:30.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536948c640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:30.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:30.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 4 set_mon_vals no callback set 2024-04-19T01:42:30.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536948c640 1 -- 172.21.15.62:0/489034154 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535c0a5ad0 con 0x7f53640a47e0 2024-04-19T01:42:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536948c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536948c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536948c640 10 monclient: dump: 2024-04-19T01:42:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:30.236 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 10 monclient: shutdown 2024-04-19T01:42:30.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:30.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 1 -- 172.21.15.62:0/489034154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53640a47e0 msgr2=0x7f53640a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:30.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 1 --2- 172.21.15.62:0/489034154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53640a47e0 0x7f53640a4bc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f535c009e50 tx=0x7f535c01c7f0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 1 -- 172.21.15.62:0/489034154 shutdown_connections 2024-04-19T01:42:30.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 1 --2- 172.21.15.62:0/489034154 >> v2:172.21.15.71:3300/0 conn(0x7f53640a5100 0x7f53641350b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 1 --2- 172.21.15.62:0/489034154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53640a47e0 0x7f53640a4bc0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 1 -- 172.21.15.62:0/489034154 >> 172.21.15.62:0/489034154 conn(0x7f53640a23d0 msgr2=0x7f53640a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:30.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 1 -- 172.21.15.62:0/489034154 shutdown_connections 2024-04-19T01:42:30.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 1 -- 172.21.15.62:0/489034154 wait complete. 2024-04-19T01:42:30.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 10 monclient: build_initial_monmap 2024-04-19T01:42:30.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 10 monclient: monmap: 2024-04-19T01:42:30.237 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:30.237 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:30.237 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:30.237 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:30.237 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:30.238 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:30.238 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:30.238 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:30.238 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:30.238 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:30.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 1 Processor -- start 2024-04-19T01:42:30.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 1 -- start start 2024-04-19T01:42:30.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 10 monclient: init 2024-04-19T01:42:30.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:30.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-19T01:42:30.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53640a47e0 0x7f53641cd980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:30.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 10 monclient(hunting): picked mon.noname-a con 0x7f53640a47e0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:30.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53640a5100 0x7f536413c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:30.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 10 monclient(hunting): picked mon.noname-c con 0x7f53640a5100 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:30.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53641355f0 0x7f536413cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:30.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 10 monclient(hunting): picked mon.noname-b con 0x7f53641355f0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:30.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:30.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f536409f1d0 con 0x7f53640a47e0 2024-04-19T01:42:30.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:30.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f536409f050 con 0x7f53641355f0 2024-04-19T01:42:30.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:30.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f536409c760 con 0x7f53640a5100 2024-04-19T01:42:30.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:30.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536c6f2640 10 monclient(hunting): authenticate will time out at 936.054710s 2024-04-19T01:42:30.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536a48e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53640a47e0 0x7f53641cd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:30.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536a48e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53640a47e0 0x7f53641cd980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36512/0 (socket says 172.21.15.62:36512) 2024-04-19T01:42:30.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536a48e640 1 -- 172.21.15.62:0/4121173577 learned_addr learned my addr 172.21.15.62:0/4121173577 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:30.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536a48e640 10 monclient(hunting): get_auth_request con 0x7f53640a47e0 auth_method 0 2024-04-19T01:42:30.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536a48e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:30.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536a48e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:30.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536a48e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:30.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f5369c8d640 1 --2- 172.21.15.62:0/4121173577 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53640a5100 0x7f536413c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:30.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f5369c8d640 10 monclient(hunting): get_auth_request con 0x7f53640a5100 auth_method 0 2024-04-19T01:42:30.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f5369c8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:30.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f5369c8d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:30.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f5369c8d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:30.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536a48e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:30.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536a48e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:30.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536a48e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:30.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f5369c8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:30.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f5369c8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:30.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f5369c8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:30.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536a48e640 10 monclient(hunting): handle_auth_done global_id 4341 payload 824 2024-04-19T01:42:30.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536a48e640 1 -- 172.21.15.62:0/4121173577 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53640a5100 msgr2=0x7f536413c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:30.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536a48e640 1 --2- 172.21.15.62:0/4121173577 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53640a5100 0x7f536413c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536a48e640 1 -- 172.21.15.62:0/4121173577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53641355f0 msgr2=0x7f536413cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:30.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536ac8f640 1 --2- 172.21.15.62:0/4121173577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53641355f0 0x7f536413cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:30.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536a48e640 1 --2- 172.21.15.62:0/4121173577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53641355f0 0x7f536413cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536a48e640 10 monclient: _finish_hunting 0 2024-04-19T01:42:30.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536a48e640 1 monclient: found mon.noname-a 2024-04-19T01:42:30.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536a48e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:30.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536a48e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:30.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.230+0000 7f536a48e640 1 -- 172.21.15.62:0/4121173577 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5364139da0 con 0x7f53640a47e0 2024-04-19T01:42:30.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.234+0000 7f536a48e640 1 -- 172.21.15.62:0/4121173577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53640a47e0 msgr2=0x7f53641cd980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:30.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.234+0000 7f536a48e640 1 --2- 172.21.15.62:0/4121173577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53640a47e0 0x7f53641cd980 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f535c0af040 tx=0x7f535c019590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:30.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.234+0000 7f5362ffd640 1 -- 172.21.15.62:0/4121173577 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535c0bc020 con 0x7f53640a47e0 2024-04-19T01:42:30.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.234+0000 7f5362ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:30.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.234+0000 7f5362ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:30.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.234+0000 7f5362ffd640 10 monclient: dump: 2024-04-19T01:42:30.244 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:30.244 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:30.244 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.244 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.244 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:30.244 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:30.244 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:30.244 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:30.244 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:30.244 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:30.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.234+0000 7f5362ffd640 10 monclient: _finish_auth 0 2024-04-19T01:42:30.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.234+0000 7f5362ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:42:30.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.234+0000 7f5362ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:30.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.234+0000 7f5362ffd640 1 -- 172.21.15.62:0/4121173577 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f535c0b88c0 con 0x7f53640a47e0 2024-04-19T01:42:30.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.234+0000 7f536c6f2640 5 monclient: authenticate success, global_id 4341 2024-04-19T01:42:30.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.234+0000 7f536c6f2640 10 monclient: _renew_subs 2024-04-19T01:42:30.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.234+0000 7f536c6f2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:30.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.234+0000 7f536c6f2640 1 -- 172.21.15.62:0/4121173577 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f536413a080 con 0x7f53640a47e0 2024-04-19T01:42:30.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.234+0000 7f5362ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:30.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.234+0000 7f5362ffd640 1 -- 172.21.15.62:0/4121173577 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535c0a5560 con 0x7f53640a47e0 2024-04-19T01:42:30.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.234+0000 7f5362ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:30.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.234+0000 7f5362ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:30.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.234+0000 7f5362ffd640 10 monclient: dump: 2024-04-19T01:42:30.245 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:30.245 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:30.246 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.246 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.246 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:30.246 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:30.246 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:30.246 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:30.246 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:30.246 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:30.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.234+0000 7f536c6f2640 10 monclient: _renew_subs 2024-04-19T01:42:30.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.234+0000 7f536c6f2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:30.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.234+0000 7f5368c8b640 4 set_mon_vals no callback set 2024-04-19T01:42:30.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.234+0000 7f536c6f2640 1 -- 172.21.15.62:0/4121173577 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f536406e590 con 0x7f53640a47e0 2024-04-19T01:42:30.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.234+0000 7f536c6f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:30.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.234+0000 7f536c6f2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:30.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.234+0000 7f536c6f2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:30.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.234+0000 7f536c6f2640 1 -- 172.21.15.62:0/4121173577 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53640692b0 con 0x7f53640a47e0 2024-04-19T01:42:30.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.234+0000 7f5362ffd640 1 -- 172.21.15.62:0/4121173577 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f535c0bf020 con 0x7f53640a47e0 2024-04-19T01:42:30.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.234+0000 7f5362ffd640 1 --2- 172.21.15.62:0/4121173577 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5350036470 0x7f5350038930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:30.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.234+0000 7f5362ffd640 1 -- 172.21.15.62:0/4121173577 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4346+0+0 (secure 0 0 0) 0x7f535c0e40b0 con 0x7f53640a47e0 2024-04-19T01:42:30.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.234+0000 7f5369c8d640 1 --2- 172.21.15.62:0/4121173577 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5350036470 0x7f5350038930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:30.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.234+0000 7f5369c8d640 10 monclient: get_auth_request con 0x7f5350036470 auth_method 0 2024-04-19T01:42:30.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.238+0000 7f5369c8d640 1 --2- 172.21.15.62:0/4121173577 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5350036470 0x7f5350038930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5364137950 tx=0x7f5358009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:30.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.238+0000 7f5362ffd640 1 -- 172.21.15.62:0/4121173577 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f535c0c6030 con 0x7f53640a47e0 2024-04-19T01:42:30.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.238+0000 7f5362ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:30.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.238+0000 7f5362ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:30.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.454+0000 7f536c6f2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:42:30.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.454+0000 7f536c6f2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:42:30.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.454+0000 7f536c6f2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:30.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.454+0000 7f536c6f2640 1 -- 172.21.15.62:0/4121173577 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f53640a4bc0 con 0x7f53640a47e0 2024-04-19T01:42:30.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.454+0000 7f5362ffd640 1 -- 172.21.15.62:0/4121173577 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f536409f7c0 con 0x7f53640a47e0 2024-04-19T01:42:30.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.454+0000 7f5362ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:42:30.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.454+0000 7f5362ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:30.455 INFO:teuthology.orchestra.run.smithi062.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","modified":"2024-04-19T01:41:38.497795Z","created":"2024-04-19T01:41:38.497795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:42:30.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.454+0000 7f536c6f2640 1 -- 172.21.15.62:0/4121173577 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5350036470 msgr2=0x7f5350038930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:30.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.454+0000 7f536c6f2640 1 --2- 172.21.15.62:0/4121173577 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5350036470 0x7f5350038930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5364137950 tx=0x7f5358009190 comp rx=0 tx=0).stop 2024-04-19T01:42:30.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.454+0000 7f536c6f2640 10 monclient: shutdown 2024-04-19T01:42:30.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.454+0000 7f536c6f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:30.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.454+0000 7f536c6f2640 1 -- 172.21.15.62:0/4121173577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53640a47e0 msgr2=0x7f53641cd980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:30.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.454+0000 7f536c6f2640 1 --2- 172.21.15.62:0/4121173577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53640a47e0 0x7f53641cd980 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f535c0af040 tx=0x7f535c019590 comp rx=0 tx=0).stop 2024-04-19T01:42:30.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.454+0000 7f536c6f2640 1 -- 172.21.15.62:0/4121173577 shutdown_connections 2024-04-19T01:42:30.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.454+0000 7f536c6f2640 1 --2- 172.21.15.62:0/4121173577 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5350036470 0x7f5350038930 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.454+0000 7f536c6f2640 1 --2- 172.21.15.62:0/4121173577 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f53640a5100 0x7f536413c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.454+0000 7f536c6f2640 1 --2- 172.21.15.62:0/4121173577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53641355f0 0x7f536413cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.454+0000 7f536c6f2640 1 --2- 172.21.15.62:0/4121173577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f53640a47e0 0x7f53641cd980 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.454+0000 7f536c6f2640 1 -- 172.21.15.62:0/4121173577 >> 172.21.15.62:0/4121173577 conn(0x7f53640a23d0 msgr2=0x7f536406f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:30.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.454+0000 7f536c6f2640 1 -- 172.21.15.62:0/4121173577 shutdown_connections 2024-04-19T01:42:30.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.454+0000 7f536c6f2640 1 -- 172.21.15.62:0/4121173577 wait complete. 2024-04-19T01:42:30.473 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-19T01:42:30.473 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-19T01:42:30.473 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-19T01:42:30.473 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-19T01:42:30.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ec5fe640 10 monclient: get_monmap_and_config 2024-04-19T01:42:30.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ec5fe640 10 monclient: build_initial_monmap 2024-04-19T01:42:30.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ec5fe640 10 monclient: monmap: 2024-04-19T01:42:30.598 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:30.598 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:30.598 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:30.602742+0000 2024-04-19T01:42:30.599 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:30.602742+0000 2024-04-19T01:42:30.599 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:30.599 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:30.599 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:30.599 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:30.599 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:30.599 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:30.599 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:30.599 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:30.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ec5fe640 1 Processor -- start 2024-04-19T01:42:30.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ec5fe640 1 -- start start 2024-04-19T01:42:30.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ec5fe640 10 monclient: init 2024-04-19T01:42:30.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ec5fe640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:30.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ec5fe640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-04-19T01:42:30.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ec5fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f41e40a47e0 0x7f41e40a4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:30.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ec5fe640 10 monclient(hunting): picked mon.noname-a con 0x7f41e40a47e0 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:30.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ec5fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f41e40a5100 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:30.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ec5fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f41e41357c0 addr v1:172.21.15.71:6789/0 2024-04-19T01:42:30.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ec5fe640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41e409f7c0 con 0x7f41e40a5100 2024-04-19T01:42:30.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ec5fe640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41e409c2d0 con 0x7f41e41357c0 2024-04-19T01:42:30.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ec5fe640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:30.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ec5fe640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f41e409ee90 con 0x7f41e40a47e0 2024-04-19T01:42:30.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ec5fe640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:30.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ec5fe640 10 monclient(hunting): authenticate will time out at 936.422254s 2024-04-19T01:42:30.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ea39a640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f41e40a47e0 0x7f41e40a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:30.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ea39a640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f41e40a47e0 0x7f41e40a4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36518/0 (socket says 172.21.15.62:36518) 2024-04-19T01:42:30.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ea39a640 1 -- 172.21.15.62:0/2738919041 learned_addr learned my addr 172.21.15.62:0/2738919041 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:30.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ea39a640 10 monclient(hunting): get_auth_request con 0x7f41e40a47e0 auth_method 0 2024-04-19T01:42:30.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41e9b99640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f41e40a5100 0x7f41e41350b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:33636/0 (socket says 172.21.15.62:33636) 2024-04-19T01:42:30.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ea39a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:30.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ea39a640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:30.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ea39a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:30.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ea39a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:30.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ea39a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:30.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ea39a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:30.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ea39a640 10 monclient(hunting): handle_auth_done global_id 4344 payload 274 2024-04-19T01:42:30.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ea39a640 1 -- 172.21.15.62:0/2738919041 >> v1:172.21.15.71:6789/0 conn(0x7f41e41357c0 legacy=0x7f41e413d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:30.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ea39a640 1 -- 172.21.15.62:0/2738919041 >> v1:172.21.15.62:6789/0 conn(0x7f41e40a5100 legacy=0x7f41e41350b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:30.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ea39a640 10 monclient: _finish_hunting 0 2024-04-19T01:42:30.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ea39a640 1 monclient: found mon.noname-a 2024-04-19T01:42:30.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ea39a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:30.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ea39a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:30.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.598+0000 7f41ea39a640 1 -- 172.21.15.62:0/2738919041 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41e409faa0 con 0x7f41e40a47e0 2024-04-19T01:42:30.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.602+0000 7f41ea39a640 1 -- 172.21.15.62:0/2738919041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41e40a47e0 msgr2=0x7f41e40a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:30.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.602+0000 7f41ea39a640 1 --2- 172.21.15.62:0/2738919041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41e40a47e0 0x7f41e40a4bc0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f41dc009e80 tx=0x7f41dc01c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aa896bfb9e46bc server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:30.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.602+0000 7f41e9398640 1 -- 172.21.15.62:0/2738919041 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41dc0a5560 con 0x7f41e40a47e0 2024-04-19T01:42:30.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.602+0000 7f41e9398640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:30.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.602+0000 7f41e9398640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:30.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.602+0000 7f41e9398640 10 monclient: dump: 2024-04-19T01:42:30.605 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:30.605 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:30.605 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.605 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.605 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:30.605 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:30.605 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:30.605 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:30.605 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:30.605 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:30.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.602+0000 7f41e9398640 10 monclient: _finish_auth 0 2024-04-19T01:42:30.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.602+0000 7f41e9398640 10 monclient: _check_auth_tickets 2024-04-19T01:42:30.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.602+0000 7f41e9398640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:30.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.602+0000 7f41ec5fe640 5 monclient: authenticate success, global_id 4344 2024-04-19T01:42:30.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.602+0000 7f41ec5fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:42:30.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.602+0000 7f41e9398640 1 -- 172.21.15.62:0/2738919041 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41dc0a5700 con 0x7f41e40a47e0 2024-04-19T01:42:30.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.602+0000 7f41e9398640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:30.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.602+0000 7f41e9398640 1 -- 172.21.15.62:0/2738919041 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41dc0a5ad0 con 0x7f41e40a47e0 2024-04-19T01:42:30.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.602+0000 7f41e9398640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:30.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.602+0000 7f41e9398640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:30.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.602+0000 7f41e9398640 10 monclient: dump: 2024-04-19T01:42:30.606 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:30.606 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:30.606 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.606 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.606 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:30.607 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:30.607 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:30.607 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:30.607 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:30.607 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:30.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.602+0000 7f41ec5fe640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:30.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.602+0000 7f41ec5fe640 4 set_mon_vals no callback set 2024-04-19T01:42:30.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.602+0000 7f41ec5fe640 10 monclient: shutdown 2024-04-19T01:42:30.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.602+0000 7f41ec5fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:30.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.602+0000 7f41ec5fe640 1 -- 172.21.15.62:0/2738919041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41e40a47e0 msgr2=0x7f41e40a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:30.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.602+0000 7f41ec5fe640 1 --2- 172.21.15.62:0/2738919041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41e40a47e0 0x7f41e40a4bc0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f41dc009e80 tx=0x7f41dc01c810 comp rx=0 tx=0).stop 2024-04-19T01:42:30.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.602+0000 7f41ec5fe640 1 -- 172.21.15.62:0/2738919041 shutdown_connections 2024-04-19T01:42:30.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.602+0000 7f41ec5fe640 1 --2- 172.21.15.62:0/2738919041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41e40a47e0 0x7f41e40a4bc0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.602+0000 7f41ec5fe640 1 -- 172.21.15.62:0/2738919041 >> 172.21.15.62:0/2738919041 conn(0x7f41e40a23d0 msgr2=0x7f41e40a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:30.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.602+0000 7f41ec5fe640 1 -- 172.21.15.62:0/2738919041 shutdown_connections 2024-04-19T01:42:30.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41ec5fe640 1 -- 172.21.15.62:0/2738919041 wait complete. 2024-04-19T01:42:30.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41ec5fe640 10 monclient: build_initial_monmap 2024-04-19T01:42:30.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41ec5fe640 10 monclient: monmap: 2024-04-19T01:42:30.613 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:30.614 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:30.614 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:30.614 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:30.614 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:30.614 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:30.614 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:30.614 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:30.614 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:30.614 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:30.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41ec5fe640 1 Processor -- start 2024-04-19T01:42:30.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41ec5fe640 1 -- start start 2024-04-19T01:42:30.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41ec5fe640 10 monclient: init 2024-04-19T01:42:30.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41ec5fe640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:30.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41ec5fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-19T01:42:30.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41ec5fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41e40a47e0 0x7f41e41cf870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:30.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41ec5fe640 10 monclient(hunting): picked mon.noname-c con 0x7f41e40a47e0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:30.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41ec5fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41e40a5100 0x7f41e4131200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:30.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41ec5fe640 10 monclient(hunting): picked mon.noname-a con 0x7f41e40a5100 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:30.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41ec5fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41e41357c0 0x7f41e413b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:30.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41ec5fe640 10 monclient(hunting): picked mon.noname-b con 0x7f41e41357c0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:30.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41ec5fe640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:30.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41ec5fe640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f41e409c8e0 con 0x7f41e40a5100 2024-04-19T01:42:30.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41ec5fe640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:30.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41ec5fe640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f41e409c760 con 0x7f41e41357c0 2024-04-19T01:42:30.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41ec5fe640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:30.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41ec5fe640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f41e4141540 con 0x7f41e40a47e0 2024-04-19T01:42:30.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41ec5fe640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:30.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41ec5fe640 10 monclient(hunting): authenticate will time out at 936.429602s 2024-04-19T01:42:30.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41e9b99640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41e40a5100 0x7f41e4131200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:30.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41e9b99640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41e40a5100 0x7f41e4131200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36526/0 (socket says 172.21.15.62:36526) 2024-04-19T01:42:30.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41e9b99640 1 -- 172.21.15.62:0/4047712909 learned_addr learned my addr 172.21.15.62:0/4047712909 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:30.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41e9b99640 10 monclient(hunting): get_auth_request con 0x7f41e40a5100 auth_method 0 2024-04-19T01:42:30.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41e9b99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:30.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41e9b99640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:30.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41e9b99640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:30.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41ea39a640 1 --2- 172.21.15.62:0/4047712909 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41e40a47e0 0x7f41e41cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:30.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41e9b99640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:30.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41e9b99640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:30.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41e9b99640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:30.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41ea39a640 10 monclient(hunting): get_auth_request con 0x7f41e40a47e0 auth_method 0 2024-04-19T01:42:30.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41ea39a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:30.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41ea39a640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:30.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41ea39a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:30.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41ea39a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:30.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41ea39a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:30.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41eab9b640 1 --2- 172.21.15.62:0/4047712909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41e41357c0 0x7f41e413b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:30.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41ea39a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:30.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41e9b99640 10 monclient(hunting): handle_auth_done global_id 4347 payload 824 2024-04-19T01:42:30.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41e9b99640 1 -- 172.21.15.62:0/4047712909 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41e40a47e0 msgr2=0x7f41e41cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:30.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41e9b99640 1 --2- 172.21.15.62:0/4047712909 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f41e40a47e0 0x7f41e41cf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41e9b99640 1 -- 172.21.15.62:0/4047712909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41e41357c0 msgr2=0x7f41e413b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:30.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41e9b99640 1 --2- 172.21.15.62:0/4047712909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41e41357c0 0x7f41e413b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41e9b99640 10 monclient: _finish_hunting 0 2024-04-19T01:42:30.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41e9b99640 1 monclient: found mon.noname-a 2024-04-19T01:42:30.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41e9b99640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:30.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41e9b99640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:30.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41e9b99640 1 -- 172.21.15.62:0/4047712909 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41e413d100 con 0x7f41e40a5100 2024-04-19T01:42:30.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41eab9b640 10 monclient: get_auth_request con 0x7f41e41357c0 auth_method 0 2024-04-19T01:42:30.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.606+0000 7f41eab9b640 1 --2- 172.21.15.62:0/4047712909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41e41357c0 0x7f41e413b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:42:30.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41e9b99640 1 -- 172.21.15.62:0/4047712909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41e40a5100 msgr2=0x7f41e4131200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:30.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41e9b99640 1 --2- 172.21.15.62:0/4047712909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41e40a5100 0x7f41e4131200 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f41d400c460 tx=0x7f41d400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:30.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41daffd640 1 -- 172.21.15.62:0/4047712909 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d4017070 con 0x7f41e40a5100 2024-04-19T01:42:30.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:30.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41daffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:30.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41daffd640 10 monclient: dump: 2024-04-19T01:42:30.619 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:30.619 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:30.619 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.619 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.620 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:30.620 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:30.620 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:30.620 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:30.620 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:30.620 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:30.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41daffd640 10 monclient: _finish_auth 0 2024-04-19T01:42:30.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41daffd640 10 monclient: _check_auth_tickets 2024-04-19T01:42:30.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:30.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41daffd640 1 -- 172.21.15.62:0/4047712909 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41d4005820 con 0x7f41e40a5100 2024-04-19T01:42:30.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41ec5fe640 5 monclient: authenticate success, global_id 4347 2024-04-19T01:42:30.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41ec5fe640 10 monclient: _renew_subs 2024-04-19T01:42:30.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41ec5fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:30.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41ec5fe640 1 -- 172.21.15.62:0/4047712909 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41e411cba0 con 0x7f41e40a5100 2024-04-19T01:42:30.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41daffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:30.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41daffd640 1 -- 172.21.15.62:0/4047712909 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d4005c10 con 0x7f41e40a5100 2024-04-19T01:42:30.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:30.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41daffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:30.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41daffd640 10 monclient: dump: 2024-04-19T01:42:30.621 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:30.621 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:30.621 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.621 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.621 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:30.621 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:30.621 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:30.622 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:30.622 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:30.622 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:30.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41e8b97640 4 set_mon_vals no callback set 2024-04-19T01:42:30.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41ec5fe640 10 monclient: _renew_subs 2024-04-19T01:42:30.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41ec5fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:30.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41ec5fe640 1 -- 172.21.15.62:0/4047712909 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41e4132010 con 0x7f41e40a5100 2024-04-19T01:42:30.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41ec5fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:30.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41ec5fe640 1 --2- 172.21.15.62:0/4047712909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41e411c540 0x7f41e413ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:30.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41ec5fe640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:30.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41ec5fe640 1 -- 172.21.15.62:0/4047712909 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f41e4072650 con 0x7f41e411c540 2024-04-19T01:42:30.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41ea39a640 1 --2- 172.21.15.62:0/4047712909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41e411c540 0x7f41e413ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:30.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41ea39a640 10 monclient: get_auth_request con 0x7f41e411c540 auth_method 0 2024-04-19T01:42:30.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41ea39a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:30.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41ea39a640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:30.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41ea39a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:30.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41ea39a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:30.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41ea39a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:30.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41ea39a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:30.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41ea39a640 10 monclient(hunting): handle_auth_done global_id 4350 payload 824 2024-04-19T01:42:30.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41ea39a640 1 -- 172.21.15.62:0/4047712909 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f41e40692b0 con 0x7f41e411c540 2024-04-19T01:42:30.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41daffd640 1 -- 172.21.15.62:0/4047712909 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f41d401a070 con 0x7f41e40a5100 2024-04-19T01:42:30.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41daffd640 1 --2- 172.21.15.62:0/4047712909 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f41c8036470 0x7f41c8038930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:30.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41daffd640 1 -- 172.21.15.62:0/4047712909 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4346+0+0 (secure 0 0 0) 0x7f41d4048750 con 0x7f41e40a5100 2024-04-19T01:42:30.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41eab9b640 1 --2- 172.21.15.62:0/4047712909 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f41c8036470 0x7f41c8038930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:30.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.610+0000 7f41eab9b640 10 monclient: get_auth_request con 0x7f41c8036470 auth_method 0 2024-04-19T01:42:30.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.614+0000 7f41ea39a640 1 -- 172.21.15.62:0/4047712909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41e411c540 msgr2=0x7f41e413ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:30.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.614+0000 7f41ea39a640 1 --2- 172.21.15.62:0/4047712909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41e411c540 0x7f41e413ba20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f41dc0ae000 tx=0x7f41dc0195b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:30.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.614+0000 7f41daffd640 1 -- 172.21.15.62:0/4047712909 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41dc0c0640 con 0x7f41e411c540 2024-04-19T01:42:30.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.614+0000 7f41daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:30.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.614+0000 7f41daffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:30.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.614+0000 7f41daffd640 10 monclient: dump: 2024-04-19T01:42:30.626 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:30.626 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:30.626 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.626 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.626 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:30.627 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:30.627 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:30.627 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:30.627 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:30.627 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:30.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.614+0000 7f41daffd640 1 -- 172.21.15.62:0/4047712909 <== mon.0 v2:172.21.15.62:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f41dc0c0640 con 0x7f41e411c540 2024-04-19T01:42:30.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.614+0000 7f41daffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:30.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.614+0000 7f41daffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:30.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.614+0000 7f41daffd640 1 -- 172.21.15.62:0/4047712909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41e411c540 msgr2=0x7f41e413ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:30.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.614+0000 7f41daffd640 1 --2- 172.21.15.62:0/4047712909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41e411c540 0x7f41e413ba20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f41dc0ae000 tx=0x7f41dc0195b0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.614+0000 7f41daffd640 1 -- 172.21.15.62:0/4047712909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41e411c540 msgr2=0x7f41e413ba20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:42:30.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.614+0000 7f41daffd640 1 --2- 172.21.15.62:0/4047712909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41e411c540 0x7f41e413ba20 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f41dc0ae000 tx=0x7f41dc0195b0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.614+0000 7f41eab9b640 1 --2- 172.21.15.62:0/4047712909 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f41c8036470 0x7f41c8038930 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f41e0003dc0 tx=0x7f41e0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:30.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.618+0000 7f41ec5fe640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:42:30.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.618+0000 7f41ec5fe640 1 --2- 172.21.15.62:0/4047712909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41e413bf60 0x7f41e406edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:30.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.618+0000 7f41ec5fe640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:30.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.618+0000 7f41ec5fe640 1 -- 172.21.15.62:0/4047712909 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f41e413c370 con 0x7f41e413bf60 2024-04-19T01:42:30.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.618+0000 7f41ea39a640 1 --2- 172.21.15.62:0/4047712909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41e413bf60 0x7f41e406edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:30.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.618+0000 7f41ea39a640 10 monclient: get_auth_request con 0x7f41e413bf60 auth_method 0 2024-04-19T01:42:30.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.618+0000 7f41ea39a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:30.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.618+0000 7f41ea39a640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:30.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.618+0000 7f41ea39a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:30.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.618+0000 7f41ea39a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:30.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.618+0000 7f41ea39a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:30.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.618+0000 7f41ea39a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:30.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.618+0000 7f41ea39a640 10 monclient(hunting): handle_auth_done global_id 4353 payload 824 2024-04-19T01:42:30.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.618+0000 7f41ea39a640 1 -- 172.21.15.62:0/4047712909 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f41e41d06f0 con 0x7f41e413bf60 2024-04-19T01:42:30.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.622+0000 7f41ea39a640 1 -- 172.21.15.62:0/4047712909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41e413bf60 msgr2=0x7f41e406edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:30.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.622+0000 7f41ea39a640 1 --2- 172.21.15.62:0/4047712909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41e413bf60 0x7f41e406edc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f41dc01c640 tx=0x7f41dc0a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:30.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.622+0000 7f41daffd640 1 -- 172.21.15.62:0/4047712909 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41dc0c9070 con 0x7f41e413bf60 2024-04-19T01:42:30.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.622+0000 7f41daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:30.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.622+0000 7f41daffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:30.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.622+0000 7f41daffd640 10 monclient: dump: 2024-04-19T01:42:30.629 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:30.629 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:30.629 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.630 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.630 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:30.630 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:30.630 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:30.630 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:30.630 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:30.630 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:30.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.622+0000 7f41daffd640 1 -- 172.21.15.62:0/4047712909 <== mon.0 v2:172.21.15.62:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7f41dc0c0c00 con 0x7f41e413bf60 2024-04-19T01:42:30.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.622+0000 7f41daffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:42:30.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.622+0000 7f41daffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:30.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.622+0000 7f41daffd640 1 -- 172.21.15.62:0/4047712909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41e413bf60 msgr2=0x7f41e406edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:30.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.622+0000 7f41daffd640 1 --2- 172.21.15.62:0/4047712909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41e413bf60 0x7f41e406edc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f41dc01c640 tx=0x7f41dc0a7000 comp rx=0 tx=0).stop 2024-04-19T01:42:30.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.622+0000 7f41daffd640 1 -- 172.21.15.62:0/4047712909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41e413bf60 msgr2=0x7f41e406edc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:42:30.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.622+0000 7f41daffd640 1 --2- 172.21.15.62:0/4047712909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41e413bf60 0x7f41e406edc0 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f41dc01c640 tx=0x7f41dc0a7000 comp rx=0 tx=0).stop 2024-04-19T01:42:30.631 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-19T01:42:30.631 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "a", 2024-04-19T01:42:30.631 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 0, 2024-04-19T01:42:30.631 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "leader", 2024-04-19T01:42:30.631 INFO:teuthology.orchestra.run.smithi062.stdout: "election_epoch": 8, 2024-04-19T01:42:30.631 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 26827, 2024-04-19T01:42:30.631 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum": [ 2024-04-19T01:42:30.631 INFO:teuthology.orchestra.run.smithi062.stdout: 0, 2024-04-19T01:42:30.631 INFO:teuthology.orchestra.run.smithi062.stdout: 1, 2024-04-19T01:42:30.631 INFO:teuthology.orchestra.run.smithi062.stdout: 2 2024-04-19T01:42:30.631 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:42:30.631 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_age": 21, 2024-04-19T01:42:30.631 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:42:30.632 INFO:teuthology.orchestra.run.smithi062.stdout: "required_con": "2450521708007866372", 2024-04-19T01:42:30.632 INFO:teuthology.orchestra.run.smithi062.stdout: "required_mon": [ 2024-04-19T01:42:30.632 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:42:30.632 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:42:30.632 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:42:30.632 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:42:30.632 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:42:30.632 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:42:30.632 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:42:30.632 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:42:30.632 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:42:30.632 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:42:30.632 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:42:30.632 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:42:30.632 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:42:30.633 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_mon": [ 2024-04-19T01:42:30.633 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:42:30.633 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:42:30.633 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:42:30.633 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:42:30.633 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:42:30.633 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:42:30.633 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:42:30.633 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:42:30.633 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:42:30.633 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:42:30.633 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:42:30.633 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:42:30.633 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.633 INFO:teuthology.orchestra.run.smithi062.stdout: "outside_quorum": [], 2024-04-19T01:42:30.633 INFO:teuthology.orchestra.run.smithi062.stdout: "extra_probe_peers": [], 2024-04-19T01:42:30.634 INFO:teuthology.orchestra.run.smithi062.stdout: "sync_provider": [], 2024-04-19T01:42:30.634 INFO:teuthology.orchestra.run.smithi062.stdout: "monmap": { 2024-04-19T01:42:30.634 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 1, 2024-04-19T01:42:30.634 INFO:teuthology.orchestra.run.smithi062.stdout: "fsid": "6466e4b6-2520-485f-95b3-9401915fbbad", 2024-04-19T01:42:30.634 INFO:teuthology.orchestra.run.smithi062.stdout: "modified": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:42:30.634 INFO:teuthology.orchestra.run.smithi062.stdout: "created": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:42:30.634 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release": 19, 2024-04-19T01:42:30.634 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release_name": "squid", 2024-04-19T01:42:30.634 INFO:teuthology.orchestra.run.smithi062.stdout: "election_strategy": 1, 2024-04-19T01:42:30.634 INFO:teuthology.orchestra.run.smithi062.stdout: "disallowed_leaders: ": "", 2024-04-19T01:42:30.634 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false, 2024-04-19T01:42:30.634 INFO:teuthology.orchestra.run.smithi062.stdout: "tiebreaker_mon": "", 2024-04-19T01:42:30.634 INFO:teuthology.orchestra.run.smithi062.stdout: "removed_ranks: ": "", 2024-04-19T01:42:30.634 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:42:30.635 INFO:teuthology.orchestra.run.smithi062.stdout: "persistent": [ 2024-04-19T01:42:30.635 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:42:30.635 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:42:30.635 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:42:30.635 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:42:30.635 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:42:30.635 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:42:30.635 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:42:30.635 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:42:30.635 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:42:30.635 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:42:30.635 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:42:30.635 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:42:30.635 INFO:teuthology.orchestra.run.smithi062.stdout: "optional": [] 2024-04-19T01:42:30.635 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.635 INFO:teuthology.orchestra.run.smithi062.stdout: "mons": [ 2024-04-19T01:42:30.636 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.636 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 0, 2024-04-19T01:42:30.636 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "a", 2024-04-19T01:42:30.636 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:42:30.636 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:42:30.636 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.636 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:42:30.636 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3300", 2024-04-19T01:42:30.636 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:42:30.636 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.636 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.636 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:42:30.636 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789", 2024-04-19T01:42:30.636 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:42:30.637 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:30.637 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:42:30.637 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.637 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789/0", 2024-04-19T01:42:30.637 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6789/0", 2024-04-19T01:42:30.637 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:42:30.637 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:42:30.637 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:42:30.637 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.638 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.638 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 1, 2024-04-19T01:42:30.638 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "b", 2024-04-19T01:42:30.638 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:42:30.638 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:42:30.638 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.639 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:42:30.639 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:3300", 2024-04-19T01:42:30.639 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:42:30.639 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.639 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.639 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:42:30.639 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789", 2024-04-19T01:42:30.639 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:42:30.639 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:30.639 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:42:30.639 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.639 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789/0", 2024-04-19T01:42:30.639 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.71:6789/0", 2024-04-19T01:42:30.640 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:42:30.640 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:42:30.640 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:42:30.640 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.640 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.640 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 2, 2024-04-19T01:42:30.640 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "c", 2024-04-19T01:42:30.640 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:42:30.640 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:42:30.640 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.640 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:42:30.640 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3301", 2024-04-19T01:42:30.640 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:42:30.640 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.641 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.641 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:42:30.641 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790", 2024-04-19T01:42:30.641 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:42:30.641 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:30.641 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:42:30.641 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.641 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790/0", 2024-04-19T01:42:30.641 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6790/0", 2024-04-19T01:42:30.641 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:42:30.641 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:42:30.641 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:42:30.641 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:30.642 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:42:30.642 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.642 INFO:teuthology.orchestra.run.smithi062.stdout: "feature_map": { 2024-04-19T01:42:30.642 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": [ 2024-04-19T01:42:30.642 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.642 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:42:30.642 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:42:30.642 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:42:30.642 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:30.642 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:42:30.642 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": [ 2024-04-19T01:42:30.642 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.642 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:42:30.643 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:42:30.643 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 2 2024-04-19T01:42:30.643 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:30.643 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:42:30.643 INFO:teuthology.orchestra.run.smithi062.stdout: "client": [ 2024-04-19T01:42:30.643 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.643 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:42:30.643 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:42:30.643 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 2 2024-04-19T01:42:30.643 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:30.643 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:42:30.643 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": [ 2024-04-19T01:42:30.643 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.643 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:42:30.643 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:42:30.643 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:42:30.644 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:30.644 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:42:30.644 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.644 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false 2024-04-19T01:42:30.644 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-19T01:42:30.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.622+0000 7f41ec5fe640 1 -- 172.21.15.62:0/4047712909 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f41c8036470 msgr2=0x7f41c8038930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:30.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.622+0000 7f41ec5fe640 1 --2- 172.21.15.62:0/4047712909 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f41c8036470 0x7f41c8038930 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f41e0003dc0 tx=0x7f41e0009040 comp rx=0 tx=0).stop 2024-04-19T01:42:30.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.622+0000 7f41ec5fe640 10 monclient: shutdown 2024-04-19T01:42:30.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.622+0000 7f41ec5fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:30.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.622+0000 7f41ec5fe640 1 -- 172.21.15.62:0/4047712909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41e40a5100 msgr2=0x7f41e4131200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:30.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.622+0000 7f41ec5fe640 1 --2- 172.21.15.62:0/4047712909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41e40a5100 0x7f41e4131200 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f41d400c460 tx=0x7f41d400c920 comp rx=0 tx=0).stop 2024-04-19T01:42:30.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.622+0000 7f41eab9b640 1 -- 172.21.15.62:0/4047712909 reap_dead start 2024-04-19T01:42:30.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.622+0000 7f41ec5fe640 1 -- 172.21.15.62:0/4047712909 shutdown_connections 2024-04-19T01:42:30.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.622+0000 7f41ec5fe640 1 --2- 172.21.15.62:0/4047712909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41e40a5100 0x7f41e4131200 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.622+0000 7f41ec5fe640 1 -- 172.21.15.62:0/4047712909 >> 172.21.15.62:0/4047712909 conn(0x7f41e40a23d0 msgr2=0x7f41e41342a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:30.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.622+0000 7f41ec5fe640 1 -- 172.21.15.62:0/4047712909 shutdown_connections 2024-04-19T01:42:30.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.622+0000 7f41ec5fe640 1 -- 172.21.15.62:0/4047712909 wait complete. 2024-04-19T01:42:30.646 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-19T01:42:30.646 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-19T01:42:30.646 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-19T01:42:30.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c483ac640 10 monclient: get_monmap_and_config 2024-04-19T01:42:30.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c483ac640 10 monclient: build_initial_monmap 2024-04-19T01:42:30.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c483ac640 10 monclient: monmap: 2024-04-19T01:42:30.762 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:30.762 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:30.762 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:30.766524+0000 2024-04-19T01:42:30.762 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:30.766524+0000 2024-04-19T01:42:30.762 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:30.763 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:30.763 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:30.763 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:30.763 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:30.763 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:30.763 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:30.763 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:30.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c483ac640 1 Processor -- start 2024-04-19T01:42:30.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c483ac640 1 -- start start 2024-04-19T01:42:30.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c483ac640 10 monclient: init 2024-04-19T01:42:30.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c483ac640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:30.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c483ac640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-04-19T01:42:30.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c483ac640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0c400a47e0 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:30.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c483ac640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0c400a4bc0 0x7f0c400a4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:30.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c483ac640 10 monclient(hunting): picked mon.noname-a con 0x7f0c400a4bc0 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:30.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c483ac640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0c40139560 0x7f0c4013b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:30.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c483ac640 10 monclient(hunting): picked mon.noname-c con 0x7f0c40139560 addr v2:172.21.15.71:3300/0 2024-04-19T01:42:30.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c483ac640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c4009e6e0 con 0x7f0c400a47e0 2024-04-19T01:42:30.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c483ac640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:30.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c483ac640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f0c4009d4e0 con 0x7f0c400a4bc0 2024-04-19T01:42:30.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c483ac640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:30.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c483ac640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f0c4009e9c0 con 0x7f0c40139560 2024-04-19T01:42:30.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c483ac640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:30.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c483ac640 10 monclient(hunting): authenticate will time out at 936.585953s 2024-04-19T01:42:30.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c46148640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f0c400a47e0 0x7f0c4009f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:33644/0 (socket says 172.21.15.62:33644) 2024-04-19T01:42:30.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c46148640 1 -- 172.21.15.62:0/209887915 learned_addr learned my addr 172.21.15.62:0/209887915 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:42:30.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c45947640 1 --2- 172.21.15.62:0/209887915 >> v2:172.21.15.62:3300/0 conn(0x7f0c400a4bc0 0x7f0c400a4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:30.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c45947640 10 monclient(hunting): get_auth_request con 0x7f0c400a4bc0 auth_method 0 2024-04-19T01:42:30.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c45947640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:30.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c45947640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:30.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c45947640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:30.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c45947640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:30.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c45947640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:30.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c45947640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:30.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c46949640 1 --2- 172.21.15.62:0/209887915 >> v2:172.21.15.71:3300/0 conn(0x7f0c40139560 0x7f0c4013b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:30.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c45947640 10 monclient(hunting): handle_auth_done global_id 4356 payload 274 2024-04-19T01:42:30.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c45947640 1 -- 172.21.15.62:0/209887915 >> v2:172.21.15.71:3300/0 conn(0x7f0c40139560 msgr2=0x7f0c4013b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:30.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c45947640 1 --2- 172.21.15.62:0/209887915 >> v2:172.21.15.71:3300/0 conn(0x7f0c40139560 0x7f0c4013b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c45947640 1 -- 172.21.15.62:0/209887915 >> v1:172.21.15.62:6789/0 conn(0x7f0c400a47e0 legacy=0x7f0c4009f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:30.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c45947640 10 monclient: _finish_hunting 0 2024-04-19T01:42:30.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c45947640 1 monclient: found mon.noname-a 2024-04-19T01:42:30.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c45947640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:30.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c45947640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:30.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c45947640 1 -- 172.21.15.62:0/209887915 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c400a5530 con 0x7f0c400a4bc0 2024-04-19T01:42:30.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c46949640 10 monclient: get_auth_request con 0x7f0c40139560 auth_method 0 2024-04-19T01:42:30.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.762+0000 7f0c46949640 1 --2- 172.21.15.62:0/209887915 >> v2:172.21.15.71:3300/0 conn(0x7f0c40139560 0x7f0c4013b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:42:30.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.766+0000 7f0c45947640 1 -- 172.21.15.62:0/209887915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c400a4bc0 msgr2=0x7f0c400a4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:30.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.766+0000 7f0c45947640 1 --2- 172.21.15.62:0/209887915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c400a4bc0 0x7f0c400a4ff0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0c400a57c0 tx=0x7f0c3001c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c69344a2c2b45aa6 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:30.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.766+0000 7f0c45146640 1 -- 172.21.15.62:0/209887915 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c300b0070 con 0x7f0c400a4bc0 2024-04-19T01:42:30.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.766+0000 7f0c45146640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:30.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.766+0000 7f0c45146640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:30.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.766+0000 7f0c45146640 10 monclient: dump: 2024-04-19T01:42:30.769 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:30.769 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:30.769 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.769 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.769 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:30.769 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:30.769 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:30.769 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:30.769 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:30.769 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:30.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.766+0000 7f0c45146640 10 monclient: _finish_auth 0 2024-04-19T01:42:30.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.766+0000 7f0c45146640 10 monclient: _check_auth_tickets 2024-04-19T01:42:30.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.766+0000 7f0c45146640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:30.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.766+0000 7f0c45146640 1 -- 172.21.15.62:0/209887915 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c300ab400 con 0x7f0c400a4bc0 2024-04-19T01:42:30.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.766+0000 7f0c45146640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:30.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.766+0000 7f0c45146640 1 -- 172.21.15.62:0/209887915 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c300ab7d0 con 0x7f0c400a4bc0 2024-04-19T01:42:30.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.766+0000 7f0c483ac640 5 monclient: authenticate success, global_id 4356 2024-04-19T01:42:30.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.766+0000 7f0c483ac640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:30.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.766+0000 7f0c483ac640 4 set_mon_vals no callback set 2024-04-19T01:42:30.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.766+0000 7f0c45146640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:30.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.766+0000 7f0c45146640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:30.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.766+0000 7f0c45146640 10 monclient: dump: 2024-04-19T01:42:30.770 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:30.771 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:30.771 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.771 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.771 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:30.771 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:30.771 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:30.771 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:30.771 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:30.771 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:30.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.766+0000 7f0c483ac640 10 monclient: shutdown 2024-04-19T01:42:30.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.766+0000 7f0c483ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:30.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.766+0000 7f0c483ac640 1 -- 172.21.15.62:0/209887915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c400a4bc0 msgr2=0x7f0c400a4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:30.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.766+0000 7f0c483ac640 1 --2- 172.21.15.62:0/209887915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c400a4bc0 0x7f0c400a4ff0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0c400a57c0 tx=0x7f0c3001c3b0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.766+0000 7f0c483ac640 1 -- 172.21.15.62:0/209887915 shutdown_connections 2024-04-19T01:42:30.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.766+0000 7f0c483ac640 1 --2- 172.21.15.62:0/209887915 >> v2:172.21.15.71:3300/0 conn(0x7f0c40139560 0x7f0c4013b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.766+0000 7f0c483ac640 1 --2- 172.21.15.62:0/209887915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c400a4bc0 0x7f0c400a4ff0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.766+0000 7f0c483ac640 1 -- 172.21.15.62:0/209887915 >> 172.21.15.62:0/209887915 conn(0x7f0c400a23d0 msgr2=0x7f0c400a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:30.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.766+0000 7f0c483ac640 1 -- 172.21.15.62:0/209887915 shutdown_connections 2024-04-19T01:42:30.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.766+0000 7f0c483ac640 1 -- 172.21.15.62:0/209887915 wait complete. 2024-04-19T01:42:30.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c483ac640 10 monclient: build_initial_monmap 2024-04-19T01:42:30.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c483ac640 10 monclient: monmap: 2024-04-19T01:42:30.777 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:30.778 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:30.778 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:30.778 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:30.778 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:30.778 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:30.778 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:30.778 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:30.778 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:30.778 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:30.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c483ac640 1 Processor -- start 2024-04-19T01:42:30.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c483ac640 1 -- start start 2024-04-19T01:42:30.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c483ac640 10 monclient: init 2024-04-19T01:42:30.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c483ac640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:30.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c483ac640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-19T01:42:30.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c483ac640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c400a47e0 0x7f0c401cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:30.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c483ac640 10 monclient(hunting): picked mon.noname-a con 0x7f0c400a47e0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:30.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c483ac640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c400a4bc0 0x7f0c40135440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:30.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c483ac640 10 monclient(hunting): picked mon.noname-c con 0x7f0c400a4bc0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:30.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c483ac640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c40139560 0x7f0c40135980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:30.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c483ac640 10 monclient(hunting): picked mon.noname-b con 0x7f0c40139560 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:30.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c483ac640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:30.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c483ac640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0c4013ea40 con 0x7f0c400a47e0 2024-04-19T01:42:30.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c483ac640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:30.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c483ac640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f0c4013e8c0 con 0x7f0c40139560 2024-04-19T01:42:30.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c483ac640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:30.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c483ac640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f0c4013ebc0 con 0x7f0c400a4bc0 2024-04-19T01:42:30.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c483ac640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:30.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c483ac640 10 monclient(hunting): authenticate will time out at 936.593318s 2024-04-19T01:42:30.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c46148640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c400a47e0 0x7f0c401cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:30.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c46148640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c400a47e0 0x7f0c401cf3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36578/0 (socket says 172.21.15.62:36578) 2024-04-19T01:42:30.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c46148640 1 -- 172.21.15.62:0/2372511632 learned_addr learned my addr 172.21.15.62:0/2372511632 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:30.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c46148640 10 monclient(hunting): get_auth_request con 0x7f0c400a47e0 auth_method 0 2024-04-19T01:42:30.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c46148640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:30.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c46148640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:30.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c46148640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:30.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c45947640 1 --2- 172.21.15.62:0/2372511632 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c400a4bc0 0x7f0c40135440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:30.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c45947640 10 monclient(hunting): get_auth_request con 0x7f0c400a4bc0 auth_method 0 2024-04-19T01:42:30.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c45947640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:30.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c45947640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:30.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c45947640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:30.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c46148640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:30.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c46148640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:30.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c46148640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:30.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c45947640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:30.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c45947640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:30.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c45947640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:30.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c46949640 1 --2- 172.21.15.62:0/2372511632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c40139560 0x7f0c40135980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:30.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c46949640 10 monclient(hunting): get_auth_request con 0x7f0c40139560 auth_method 0 2024-04-19T01:42:30.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c46949640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:30.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c46949640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:30.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c46949640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:30.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c46148640 10 monclient(hunting): handle_auth_done global_id 4359 payload 824 2024-04-19T01:42:30.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c46148640 1 -- 172.21.15.62:0/2372511632 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c400a4bc0 msgr2=0x7f0c40135440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:30.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c46148640 1 --2- 172.21.15.62:0/2372511632 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c400a4bc0 0x7f0c40135440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c46148640 1 -- 172.21.15.62:0/2372511632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c40139560 msgr2=0x7f0c40135980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:30.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c46148640 1 --2- 172.21.15.62:0/2372511632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c40139560 0x7f0c40135980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c46148640 10 monclient: _finish_hunting 0 2024-04-19T01:42:30.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c46148640 1 monclient: found mon.noname-a 2024-04-19T01:42:30.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c46148640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:30.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c46148640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:30.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.770+0000 7f0c46148640 1 -- 172.21.15.62:0/2372511632 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c401228f0 con 0x7f0c400a47e0 2024-04-19T01:42:30.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c46148640 1 -- 172.21.15.62:0/2372511632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c400a47e0 msgr2=0x7f0c401cf3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:30.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c46148640 1 --2- 172.21.15.62:0/2372511632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c400a47e0 0x7f0c401cf3d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c00c470 tx=0x7f0c3c00c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:30.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c36ffd640 1 -- 172.21.15.62:0/2372511632 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c3c007070 con 0x7f0c400a47e0 2024-04-19T01:42:30.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:30.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c36ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:30.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c36ffd640 10 monclient: dump: 2024-04-19T01:42:30.782 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:30.782 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:30.782 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.783 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.783 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:30.783 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:30.783 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:30.783 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:30.783 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:30.783 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:30.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c36ffd640 10 monclient: _finish_auth 0 2024-04-19T01:42:30.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c36ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:42:30.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:30.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c36ffd640 1 -- 172.21.15.62:0/2372511632 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c3c007210 con 0x7f0c400a47e0 2024-04-19T01:42:30.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c36ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:30.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c36ffd640 1 -- 172.21.15.62:0/2372511632 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c3c007600 con 0x7f0c400a47e0 2024-04-19T01:42:30.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:30.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c36ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:30.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c36ffd640 10 monclient: dump: 2024-04-19T01:42:30.784 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:30.784 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:30.784 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.784 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.784 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:30.784 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:30.784 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:30.784 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:30.784 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:30.784 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:30.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c44945640 4 set_mon_vals no callback set 2024-04-19T01:42:30.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c483ac640 5 monclient: authenticate success, global_id 4359 2024-04-19T01:42:30.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c483ac640 10 monclient: _renew_subs 2024-04-19T01:42:30.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c483ac640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:30.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c483ac640 1 -- 172.21.15.62:0/2372511632 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c40122bd0 con 0x7f0c400a47e0 2024-04-19T01:42:30.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c483ac640 10 monclient: _renew_subs 2024-04-19T01:42:30.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c483ac640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:30.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c483ac640 1 -- 172.21.15.62:0/2372511632 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c4009e050 con 0x7f0c400a47e0 2024-04-19T01:42:30.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c483ac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:30.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c483ac640 1 --2- 172.21.15.62:0/2372511632 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c4011ec40 0x7f0c4011f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:30.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c483ac640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:30.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c483ac640 1 -- 172.21.15.62:0/2372511632 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f0c400692b0 con 0x7f0c4011ec40 2024-04-19T01:42:30.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c45947640 1 --2- 172.21.15.62:0/2372511632 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c4011ec40 0x7f0c4011f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:30.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c45947640 10 monclient: get_auth_request con 0x7f0c4011ec40 auth_method 0 2024-04-19T01:42:30.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c45947640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:30.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c45947640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:30.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c45947640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:30.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c45947640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:30.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c45947640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:30.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c45947640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:30.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c36ffd640 1 -- 172.21.15.62:0/2372511632 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f0c3c01b400 con 0x7f0c400a47e0 2024-04-19T01:42:30.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c45947640 10 monclient(hunting): handle_auth_done global_id 4355 payload 824 2024-04-19T01:42:30.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c45947640 1 -- 172.21.15.62:0/2372511632 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0c400723c0 con 0x7f0c4011ec40 2024-04-19T01:42:30.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c36ffd640 1 --2- 172.21.15.62:0/2372511632 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0c24036470 0x7f0c24038930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:30.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c36ffd640 1 -- 172.21.15.62:0/2372511632 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4346+0+0 (secure 0 0 0) 0x7f0c3c049060 con 0x7f0c400a47e0 2024-04-19T01:42:30.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c46949640 1 --2- 172.21.15.62:0/2372511632 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0c24036470 0x7f0c24038930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:30.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.774+0000 7f0c46949640 10 monclient: get_auth_request con 0x7f0c24036470 auth_method 0 2024-04-19T01:42:30.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.778+0000 7f0c45947640 1 -- 172.21.15.62:0/2372511632 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c4011ec40 msgr2=0x7f0c4011f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:30.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.778+0000 7f0c45947640 1 --2- 172.21.15.62:0/2372511632 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c4011ec40 0x7f0c4011f020 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0c300ad000 tx=0x7f0c3001abf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:30.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.778+0000 7f0c36ffd640 1 -- 172.21.15.62:0/2372511632 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c300bb070 con 0x7f0c4011ec40 2024-04-19T01:42:30.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.778+0000 7f0c36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:30.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.778+0000 7f0c36ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:30.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.778+0000 7f0c36ffd640 10 monclient: dump: 2024-04-19T01:42:30.789 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:30.789 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:30.789 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.789 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.789 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:30.789 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:30.789 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:30.789 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:30.789 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:30.789 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:30.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.778+0000 7f0c36ffd640 1 -- 172.21.15.62:0/2372511632 <== mon.2 v2:172.21.15.62:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0c300b6650 con 0x7f0c4011ec40 2024-04-19T01:42:30.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.778+0000 7f0c36ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:30.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.778+0000 7f0c36ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:30.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.778+0000 7f0c36ffd640 1 -- 172.21.15.62:0/2372511632 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c4011ec40 msgr2=0x7f0c4011f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:30.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.778+0000 7f0c36ffd640 1 --2- 172.21.15.62:0/2372511632 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c4011ec40 0x7f0c4011f020 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0c300ad000 tx=0x7f0c3001abf0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.778+0000 7f0c36ffd640 1 -- 172.21.15.62:0/2372511632 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c4011ec40 msgr2=0x7f0c4011f020 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:42:30.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.778+0000 7f0c36ffd640 1 --2- 172.21.15.62:0/2372511632 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c4011ec40 0x7f0c4011f020 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0c300ad000 tx=0x7f0c3001abf0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.778+0000 7f0c46949640 1 --2- 172.21.15.62:0/2372511632 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0c24036470 0x7f0c24038930 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0c40121940 tx=0x7f0c38007540 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:30.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.782+0000 7f0c483ac640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:42:30.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.782+0000 7f0c483ac640 1 --2- 172.21.15.62:0/2372511632 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c4011f560 0x7f0c40139a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:30.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.782+0000 7f0c483ac640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:30.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.782+0000 7f0c483ac640 1 -- 172.21.15.62:0/2372511632 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f0c40122570 con 0x7f0c4011f560 2024-04-19T01:42:30.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.782+0000 7f0c45947640 1 --2- 172.21.15.62:0/2372511632 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c4011f560 0x7f0c40139a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:30.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.782+0000 7f0c45947640 10 monclient: get_auth_request con 0x7f0c4011f560 auth_method 0 2024-04-19T01:42:30.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.782+0000 7f0c45947640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:30.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.782+0000 7f0c45947640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:30.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.782+0000 7f0c45947640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:30.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.782+0000 7f0c45947640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:30.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.782+0000 7f0c45947640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:30.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.782+0000 7f0c45947640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:30.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.782+0000 7f0c45947640 10 monclient(hunting): handle_auth_done global_id 4358 payload 824 2024-04-19T01:42:30.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.782+0000 7f0c45947640 1 -- 172.21.15.62:0/2372511632 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0c40122330 con 0x7f0c4011f560 2024-04-19T01:42:30.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.786+0000 7f0c45947640 1 -- 172.21.15.62:0/2372511632 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c4011f560 msgr2=0x7f0c40139a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:30.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.786+0000 7f0c45947640 1 --2- 172.21.15.62:0/2372511632 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c4011f560 0x7f0c40139a40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0c3001cc50 tx=0x7f0c30019150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:30.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.786+0000 7f0c36ffd640 1 -- 172.21.15.62:0/2372511632 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c300bb070 con 0x7f0c4011f560 2024-04-19T01:42:30.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.786+0000 7f0c36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:30.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.786+0000 7f0c36ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:30.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.786+0000 7f0c36ffd640 10 monclient: dump: 2024-04-19T01:42:30.791 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:30.791 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:30.791 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.791 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.791 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:30.791 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:30.792 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:30.792 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:30.792 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:30.792 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:30.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.786+0000 7f0c36ffd640 1 -- 172.21.15.62:0/2372511632 <== mon.2 v2:172.21.15.62:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4716 (secure 0 0 0) 0x7f0c30003820 con 0x7f0c4011f560 2024-04-19T01:42:30.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.786+0000 7f0c36ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:42:30.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.786+0000 7f0c36ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:30.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.786+0000 7f0c36ffd640 1 -- 172.21.15.62:0/2372511632 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c4011f560 msgr2=0x7f0c40139a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:30.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.786+0000 7f0c36ffd640 1 --2- 172.21.15.62:0/2372511632 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c4011f560 0x7f0c40139a40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0c3001cc50 tx=0x7f0c30019150 comp rx=0 tx=0).stop 2024-04-19T01:42:30.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.786+0000 7f0c36ffd640 1 -- 172.21.15.62:0/2372511632 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c4011f560 msgr2=0x7f0c40139a40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:42:30.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.786+0000 7f0c36ffd640 1 --2- 172.21.15.62:0/2372511632 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c4011f560 0x7f0c40139a40 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0c3001cc50 tx=0x7f0c30019150 comp rx=0 tx=0).stop 2024-04-19T01:42:30.792 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-19T01:42:30.793 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "c", 2024-04-19T01:42:30.793 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 2, 2024-04-19T01:42:30.793 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "peon", 2024-04-19T01:42:30.793 INFO:teuthology.orchestra.run.smithi062.stdout: "election_epoch": 8, 2024-04-19T01:42:30.793 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 26991, 2024-04-19T01:42:30.793 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum": [ 2024-04-19T01:42:30.793 INFO:teuthology.orchestra.run.smithi062.stdout: 0, 2024-04-19T01:42:30.793 INFO:teuthology.orchestra.run.smithi062.stdout: 1, 2024-04-19T01:42:30.793 INFO:teuthology.orchestra.run.smithi062.stdout: 2 2024-04-19T01:42:30.793 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:42:30.793 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_age": 21, 2024-04-19T01:42:30.793 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:42:30.793 INFO:teuthology.orchestra.run.smithi062.stdout: "required_con": "2450521708007866372", 2024-04-19T01:42:30.793 INFO:teuthology.orchestra.run.smithi062.stdout: "required_mon": [ 2024-04-19T01:42:30.793 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:42:30.794 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:42:30.794 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:42:30.794 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:42:30.794 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:42:30.794 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:42:30.794 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:42:30.794 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:42:30.794 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:42:30.794 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:42:30.794 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:42:30.794 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:42:30.794 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:42:30.794 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_mon": [ 2024-04-19T01:42:30.794 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:42:30.794 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:42:30.795 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:42:30.795 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:42:30.795 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:42:30.795 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:42:30.795 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:42:30.795 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:42:30.795 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:42:30.795 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:42:30.795 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:42:30.795 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:42:30.795 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.795 INFO:teuthology.orchestra.run.smithi062.stdout: "outside_quorum": [], 2024-04-19T01:42:30.795 INFO:teuthology.orchestra.run.smithi062.stdout: "extra_probe_peers": [], 2024-04-19T01:42:30.795 INFO:teuthology.orchestra.run.smithi062.stdout: "sync_provider": [], 2024-04-19T01:42:30.795 INFO:teuthology.orchestra.run.smithi062.stdout: "monmap": { 2024-04-19T01:42:30.795 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 1, 2024-04-19T01:42:30.796 INFO:teuthology.orchestra.run.smithi062.stdout: "fsid": "6466e4b6-2520-485f-95b3-9401915fbbad", 2024-04-19T01:42:30.796 INFO:teuthology.orchestra.run.smithi062.stdout: "modified": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:42:30.796 INFO:teuthology.orchestra.run.smithi062.stdout: "created": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:42:30.796 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release": 19, 2024-04-19T01:42:30.796 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release_name": "squid", 2024-04-19T01:42:30.796 INFO:teuthology.orchestra.run.smithi062.stdout: "election_strategy": 1, 2024-04-19T01:42:30.796 INFO:teuthology.orchestra.run.smithi062.stdout: "disallowed_leaders: ": "", 2024-04-19T01:42:30.796 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false, 2024-04-19T01:42:30.796 INFO:teuthology.orchestra.run.smithi062.stdout: "tiebreaker_mon": "", 2024-04-19T01:42:30.796 INFO:teuthology.orchestra.run.smithi062.stdout: "removed_ranks: ": "", 2024-04-19T01:42:30.796 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:42:30.796 INFO:teuthology.orchestra.run.smithi062.stdout: "persistent": [ 2024-04-19T01:42:30.796 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:42:30.796 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:42:30.796 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:42:30.796 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:42:30.797 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:42:30.797 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:42:30.797 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:42:30.797 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:42:30.797 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:42:30.797 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:42:30.797 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:42:30.797 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:42:30.797 INFO:teuthology.orchestra.run.smithi062.stdout: "optional": [] 2024-04-19T01:42:30.797 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.797 INFO:teuthology.orchestra.run.smithi062.stdout: "mons": [ 2024-04-19T01:42:30.797 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.797 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 0, 2024-04-19T01:42:30.797 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "a", 2024-04-19T01:42:30.797 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:42:30.798 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:42:30.798 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.798 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:42:30.798 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3300", 2024-04-19T01:42:30.798 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:42:30.798 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.798 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.798 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:42:30.798 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789", 2024-04-19T01:42:30.798 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:42:30.798 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:30.798 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:42:30.798 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.798 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789/0", 2024-04-19T01:42:30.798 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6789/0", 2024-04-19T01:42:30.798 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:42:30.798 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:42:30.799 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:42:30.799 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.799 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.799 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 1, 2024-04-19T01:42:30.799 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "b", 2024-04-19T01:42:30.799 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:42:30.799 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:42:30.799 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.799 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:42:30.799 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:3300", 2024-04-19T01:42:30.799 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:42:30.799 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.799 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.799 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:42:30.799 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789", 2024-04-19T01:42:30.799 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:42:30.799 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:30.800 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:42:30.800 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.800 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789/0", 2024-04-19T01:42:30.800 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.71:6789/0", 2024-04-19T01:42:30.800 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:42:30.800 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:42:30.800 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:42:30.800 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.800 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.800 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 2, 2024-04-19T01:42:30.800 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "c", 2024-04-19T01:42:30.800 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:42:30.800 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:42:30.800 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.800 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:42:30.800 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3301", 2024-04-19T01:42:30.801 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:42:30.801 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.801 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.801 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:42:30.801 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790", 2024-04-19T01:42:30.801 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:42:30.801 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:30.801 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:42:30.801 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.801 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790/0", 2024-04-19T01:42:30.801 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6790/0", 2024-04-19T01:42:30.801 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:42:30.801 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:42:30.801 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:42:30.802 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:30.802 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:42:30.802 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.802 INFO:teuthology.orchestra.run.smithi062.stdout: "feature_map": { 2024-04-19T01:42:30.802 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": [ 2024-04-19T01:42:30.802 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.802 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:42:30.802 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:42:30.802 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:42:30.802 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:30.802 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:42:30.802 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": [ 2024-04-19T01:42:30.802 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.802 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:42:30.802 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:42:30.803 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:42:30.803 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:30.803 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:42:30.803 INFO:teuthology.orchestra.run.smithi062.stdout: "client": [ 2024-04-19T01:42:30.803 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.803 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:42:30.803 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:42:30.803 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 2 2024-04-19T01:42:30.803 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:30.803 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:42:30.803 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.803 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false 2024-04-19T01:42:30.803 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-19T01:42:30.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.786+0000 7f0c483ac640 1 -- 172.21.15.62:0/2372511632 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0c24036470 msgr2=0x7f0c24038930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:30.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.786+0000 7f0c483ac640 1 --2- 172.21.15.62:0/2372511632 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0c24036470 0x7f0c24038930 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0c40121940 tx=0x7f0c38007540 comp rx=0 tx=0).stop 2024-04-19T01:42:30.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.786+0000 7f0c483ac640 10 monclient: shutdown 2024-04-19T01:42:30.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.786+0000 7f0c483ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:30.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.786+0000 7f0c483ac640 1 -- 172.21.15.62:0/2372511632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c400a47e0 msgr2=0x7f0c401cf3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:30.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.786+0000 7f0c483ac640 1 --2- 172.21.15.62:0/2372511632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c400a47e0 0x7f0c401cf3d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c00c470 tx=0x7f0c3c00c930 comp rx=0 tx=0).stop 2024-04-19T01:42:30.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.786+0000 7f0c46949640 1 -- 172.21.15.62:0/2372511632 reap_dead start 2024-04-19T01:42:30.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.786+0000 7f0c483ac640 1 -- 172.21.15.62:0/2372511632 shutdown_connections 2024-04-19T01:42:30.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.786+0000 7f0c483ac640 1 --2- 172.21.15.62:0/2372511632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c400a47e0 0x7f0c401cf3d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.786+0000 7f0c483ac640 1 -- 172.21.15.62:0/2372511632 >> 172.21.15.62:0/2372511632 conn(0x7f0c400a23d0 msgr2=0x7f0c400a1710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:30.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.786+0000 7f0c483ac640 1 -- 172.21.15.62:0/2372511632 shutdown_connections 2024-04-19T01:42:30.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.786+0000 7f0c483ac640 1 -- 172.21.15.62:0/2372511632 wait complete. 2024-04-19T01:42:30.806 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-19T01:42:30.806 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-19T01:42:30.806 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T01:42:30.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c223ad640 10 monclient: get_monmap_and_config 2024-04-19T01:42:30.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c223ad640 10 monclient: build_initial_monmap 2024-04-19T01:42:30.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c223ad640 10 monclient: monmap: 2024-04-19T01:42:30.922 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:30.922 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:30.922 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:30.926337+0000 2024-04-19T01:42:30.922 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:30.926337+0000 2024-04-19T01:42:30.922 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:30.922 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:30.922 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:30.922 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:30.923 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:30.923 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:30.923 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:30.923 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:30.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c223ad640 1 Processor -- start 2024-04-19T01:42:30.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c223ad640 1 -- start start 2024-04-19T01:42:30.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c223ad640 10 monclient: init 2024-04-19T01:42:30.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c223ad640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:30.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c223ad640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-04-19T01:42:30.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c223ad640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0c1c137780 0x7f0c1c137b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:30.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c223ad640 10 monclient(hunting): picked mon.noname-a con 0x7f0c1c137780 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:30.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c223ad640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0c1c1380a0 0x7f0c1c0a45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:30.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c223ad640 10 monclient(hunting): picked mon.noname-c con 0x7f0c1c1380a0 addr v2:172.21.15.71:3300/0 2024-04-19T01:42:30.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c223ad640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c1c0a60b0 0x7f0c1c0a4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:30.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c223ad640 10 monclient(hunting): picked mon.noname-b con 0x7f0c1c0a60b0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:30.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c223ad640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:30.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c223ad640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f0c1c09bd20 con 0x7f0c1c137780 2024-04-19T01:42:30.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c223ad640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:30.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c223ad640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f0c1c09c2e0 con 0x7f0c1c1380a0 2024-04-19T01:42:30.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c223ad640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:30.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c223ad640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f0c1c09c000 con 0x7f0c1c0a60b0 2024-04-19T01:42:30.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c223ad640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:30.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c223ad640 10 monclient(hunting): authenticate will time out at 936.745816s 2024-04-19T01:42:30.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c1bfff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0c1c137780 0x7f0c1c137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:30.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c1bfff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0c1c137780 0x7f0c1c137b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36580/0 (socket says 172.21.15.62:36580) 2024-04-19T01:42:30.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c2094a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c1c0a60b0 0x7f0c1c0a4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:30.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c1bfff640 1 -- 172.21.15.62:0/3863624058 learned_addr learned my addr 172.21.15.62:0/3863624058 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:30.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c1bfff640 10 monclient(hunting): get_auth_request con 0x7f0c1c137780 auth_method 0 2024-04-19T01:42:30.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:30.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c1bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:30.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:30.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c2094a640 10 monclient(hunting): get_auth_request con 0x7f0c1c0a60b0 auth_method 0 2024-04-19T01:42:30.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c2094a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:30.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c2094a640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:30.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c2094a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:30.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:30.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:30.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:30.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c2094a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:30.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c2094a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:30.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c2094a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:30.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c1b7fe640 1 --2- 172.21.15.62:0/3863624058 >> v2:172.21.15.71:3300/0 conn(0x7f0c1c1380a0 0x7f0c1c0a45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:30.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c1b7fe640 10 monclient(hunting): get_auth_request con 0x7f0c1c1380a0 auth_method 0 2024-04-19T01:42:30.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:30.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c1b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:30.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:30.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c2094a640 10 monclient(hunting): handle_auth_done global_id 4361 payload 274 2024-04-19T01:42:30.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c2094a640 1 -- 172.21.15.62:0/3863624058 >> v2:172.21.15.71:3300/0 conn(0x7f0c1c1380a0 msgr2=0x7f0c1c0a45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:30.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c2094a640 1 --2- 172.21.15.62:0/3863624058 >> v2:172.21.15.71:3300/0 conn(0x7f0c1c1380a0 0x7f0c1c0a45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c2094a640 1 -- 172.21.15.62:0/3863624058 >> v2:172.21.15.62:3300/0 conn(0x7f0c1c137780 msgr2=0x7f0c1c137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:30.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c2094a640 1 --2- 172.21.15.62:0/3863624058 >> v2:172.21.15.62:3300/0 conn(0x7f0c1c137780 0x7f0c1c137b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c2094a640 10 monclient: _finish_hunting 0 2024-04-19T01:42:30.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c2094a640 1 monclient: found mon.noname-b 2024-04-19T01:42:30.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c2094a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:30.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c2094a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:42:30.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c2094a640 1 -- 172.21.15.62:0/3863624058 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c1c073210 con 0x7f0c1c0a60b0 2024-04-19T01:42:30.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.922+0000 7f0c1bfff640 1 --2- 172.21.15.62:0/3863624058 >> v2:172.21.15.62:3300/0 conn(0x7f0c1c137780 0x7f0c1c137b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:42:30.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.926+0000 7f0c2094a640 1 -- 172.21.15.62:0/3863624058 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c1c0a60b0 msgr2=0x7f0c1c0a4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:30.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.926+0000 7f0c2094a640 1 --2- 172.21.15.62:0/3863624058 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c1c0a60b0 0x7f0c1c0a4b10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c0a74f0 tx=0x7f0c1401c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=500ca9c51e8a3aa7 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:30.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.926+0000 7f0c1affd640 1 -- 172.21.15.62:0/3863624058 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c140b2070 con 0x7f0c1c0a60b0 2024-04-19T01:42:30.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.926+0000 7f0c1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:30.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.926+0000 7f0c1affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:42:30.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.926+0000 7f0c1affd640 10 monclient: dump: 2024-04-19T01:42:30.933 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:30.933 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:30.933 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.933 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.933 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:30.933 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:30.933 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:30.933 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:30.933 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:30.933 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:30.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.926+0000 7f0c1affd640 10 monclient: _finish_auth 0 2024-04-19T01:42:30.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.926+0000 7f0c1affd640 10 monclient: _check_auth_tickets 2024-04-19T01:42:30.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.926+0000 7f0c1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:30.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.926+0000 7f0c223ad640 5 monclient: authenticate success, global_id 4361 2024-04-19T01:42:30.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.926+0000 7f0c223ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:42:30.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.926+0000 7f0c1affd640 1 -- 172.21.15.62:0/3863624058 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c140a5780 con 0x7f0c1c0a60b0 2024-04-19T01:42:30.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.926+0000 7f0c1affd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:30.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.926+0000 7f0c223ad640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:30.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.926+0000 7f0c223ad640 4 set_mon_vals no callback set 2024-04-19T01:42:30.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.926+0000 7f0c1affd640 1 -- 172.21.15.62:0/3863624058 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c140a5b50 con 0x7f0c1c0a60b0 2024-04-19T01:42:30.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.926+0000 7f0c1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:30.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.926+0000 7f0c1affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:30.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.926+0000 7f0c1affd640 10 monclient: dump: 2024-04-19T01:42:30.934 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:30.934 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:30.935 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.935 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.935 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:30.935 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:30.935 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:30.935 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:30.935 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:30.935 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:30.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.926+0000 7f0c223ad640 10 monclient: shutdown 2024-04-19T01:42:30.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.926+0000 7f0c223ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:30.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.926+0000 7f0c223ad640 1 -- 172.21.15.62:0/3863624058 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c1c0a60b0 msgr2=0x7f0c1c0a4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:30.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.926+0000 7f0c223ad640 1 --2- 172.21.15.62:0/3863624058 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c1c0a60b0 0x7f0c1c0a4b10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c0a74f0 tx=0x7f0c1401c7f0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.926+0000 7f0c223ad640 1 -- 172.21.15.62:0/3863624058 shutdown_connections 2024-04-19T01:42:30.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.926+0000 7f0c223ad640 1 --2- 172.21.15.62:0/3863624058 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c1c0a60b0 0x7f0c1c0a4b10 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.926+0000 7f0c223ad640 1 --2- 172.21.15.62:0/3863624058 >> v2:172.21.15.71:3300/0 conn(0x7f0c1c1380a0 0x7f0c1c0a45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.926+0000 7f0c223ad640 1 --2- 172.21.15.62:0/3863624058 >> v2:172.21.15.62:3300/0 conn(0x7f0c1c137780 0x7f0c1c137b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.926+0000 7f0c223ad640 1 -- 172.21.15.62:0/3863624058 >> 172.21.15.62:0/3863624058 conn(0x7f0c1c0aa1d0 msgr2=0x7f0c1c0a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:30.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.926+0000 7f0c223ad640 1 -- 172.21.15.62:0/3863624058 shutdown_connections 2024-04-19T01:42:30.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.926+0000 7f0c223ad640 1 -- 172.21.15.62:0/3863624058 wait complete. 2024-04-19T01:42:30.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c223ad640 10 monclient: build_initial_monmap 2024-04-19T01:42:30.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c223ad640 10 monclient: monmap: 2024-04-19T01:42:30.936 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:30.936 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:30.936 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:30.936 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:30.936 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:30.936 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:30.936 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:30.936 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:30.936 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:30.936 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:30.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c223ad640 1 Processor -- start 2024-04-19T01:42:30.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c223ad640 1 -- start start 2024-04-19T01:42:30.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c223ad640 10 monclient: init 2024-04-19T01:42:30.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c223ad640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:30.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c223ad640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-19T01:42:30.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c223ad640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c1c0a60b0 0x7f0c1c120660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:30.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c223ad640 10 monclient(hunting): picked mon.noname-a con 0x7f0c1c0a60b0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:30.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c223ad640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c1c137780 0x7f0c1c0a0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:30.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c223ad640 10 monclient(hunting): picked mon.noname-b con 0x7f0c1c137780 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:30.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c223ad640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c1c1380a0 0x7f0c1c1364c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:30.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c223ad640 10 monclient(hunting): picked mon.noname-c con 0x7f0c1c1380a0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:30.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c223ad640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:30.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c223ad640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1c141160 con 0x7f0c1c0a60b0 2024-04-19T01:42:30.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c223ad640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:30.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c223ad640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1c0a5460 con 0x7f0c1c137780 2024-04-19T01:42:30.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c223ad640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:30.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c223ad640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f0c1c1412e0 con 0x7f0c1c1380a0 2024-04-19T01:42:30.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c223ad640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:30.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c223ad640 10 monclient(hunting): authenticate will time out at 936.753122s 2024-04-19T01:42:30.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c1bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c1c0a60b0 0x7f0c1c120660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:30.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c2094a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c1c1380a0 0x7f0c1c1364c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:30.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c1bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c1c0a60b0 0x7f0c1c120660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36590/0 (socket says 172.21.15.62:36590) 2024-04-19T01:42:30.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c1bfff640 1 -- 172.21.15.62:0/2255897224 learned_addr learned my addr 172.21.15.62:0/2255897224 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:30.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c1bfff640 10 monclient(hunting): get_auth_request con 0x7f0c1c0a60b0 auth_method 0 2024-04-19T01:42:30.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:30.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c1bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:30.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:30.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c2094a640 10 monclient(hunting): get_auth_request con 0x7f0c1c1380a0 auth_method 0 2024-04-19T01:42:30.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c2094a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:30.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c2094a640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:30.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c2094a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:30.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:30.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:30.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:30.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c2094a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:30.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c2094a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:30.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c2094a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:30.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c1b7fe640 1 --2- 172.21.15.62:0/2255897224 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c1c137780 0x7f0c1c0a0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:30.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c1b7fe640 10 monclient(hunting): get_auth_request con 0x7f0c1c137780 auth_method 0 2024-04-19T01:42:30.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:30.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c1b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:30.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:30.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c1bfff640 10 monclient(hunting): handle_auth_done global_id 4365 payload 824 2024-04-19T01:42:30.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c1bfff640 1 -- 172.21.15.62:0/2255897224 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c1c1380a0 msgr2=0x7f0c1c1364c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:30.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c1bfff640 1 --2- 172.21.15.62:0/2255897224 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0c1c1380a0 0x7f0c1c1364c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c1bfff640 1 -- 172.21.15.62:0/2255897224 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c1c137780 msgr2=0x7f0c1c0a0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:30.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c1bfff640 1 --2- 172.21.15.62:0/2255897224 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c1c137780 0x7f0c1c0a0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c1bfff640 10 monclient: _finish_hunting 0 2024-04-19T01:42:30.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c1bfff640 1 monclient: found mon.noname-a 2024-04-19T01:42:30.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:30.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c1bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:42:30.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c1bfff640 1 -- 172.21.15.62:0/2255897224 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c1c121e60 con 0x7f0c1c0a60b0 2024-04-19T01:42:30.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c1bfff640 1 -- 172.21.15.62:0/2255897224 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c1c0a60b0 msgr2=0x7f0c1c120660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:30.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.930+0000 7f0c1bfff640 1 --2- 172.21.15.62:0/2255897224 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c1c0a60b0 0x7f0c1c120660 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c009fd0 tx=0x7f0c0c00ee10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:30.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c18ff9640 1 -- 172.21.15.62:0/2255897224 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c00cc10 con 0x7f0c1c0a60b0 2024-04-19T01:42:30.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:30.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c18ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:42:30.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c18ff9640 10 monclient: dump: 2024-04-19T01:42:30.943 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:30.943 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:30.943 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.943 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.943 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:30.944 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:30.944 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:30.944 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:30.944 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:30.944 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:30.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c18ff9640 10 monclient: _finish_auth 0 2024-04-19T01:42:30.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c18ff9640 10 monclient: _check_auth_tickets 2024-04-19T01:42:30.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c18ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:30.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c18ff9640 1 -- 172.21.15.62:0/2255897224 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c0c00cdb0 con 0x7f0c1c0a60b0 2024-04-19T01:42:30.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c18ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:30.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c18ff9640 1 -- 172.21.15.62:0/2255897224 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c004250 con 0x7f0c1c0a60b0 2024-04-19T01:42:30.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c223ad640 5 monclient: authenticate success, global_id 4365 2024-04-19T01:42:30.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c223ad640 10 monclient: _renew_subs 2024-04-19T01:42:30.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c223ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:30.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c223ad640 1 -- 172.21.15.62:0/2255897224 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c1c13de80 con 0x7f0c1c0a60b0 2024-04-19T01:42:30.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c1a7fc640 4 set_mon_vals no callback set 2024-04-19T01:42:30.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:30.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c18ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:42:30.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c18ff9640 10 monclient: dump: 2024-04-19T01:42:30.945 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:30.945 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:30.945 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.945 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.945 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:30.945 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:30.945 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:30.945 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:30.945 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:30.945 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:30.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c223ad640 10 monclient: _renew_subs 2024-04-19T01:42:30.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c223ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:42:30.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c223ad640 1 -- 172.21.15.62:0/2255897224 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c1c1215c0 con 0x7f0c1c0a60b0 2024-04-19T01:42:30.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c223ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:30.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c223ad640 1 --2- 172.21.15.62:0/2255897224 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c1c136a00 0x7f0c1c1cf470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:30.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c223ad640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:30.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c223ad640 1 -- 172.21.15.62:0/2255897224 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1c0a34b0 con 0x7f0c1c136a00 2024-04-19T01:42:30.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c18ff9640 1 -- 172.21.15.62:0/2255897224 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f0c0c003620 con 0x7f0c1c0a60b0 2024-04-19T01:42:30.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c1b7fe640 1 --2- 172.21.15.62:0/2255897224 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c1c136a00 0x7f0c1c1cf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:30.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c1b7fe640 10 monclient: get_auth_request con 0x7f0c1c136a00 auth_method 0 2024-04-19T01:42:30.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:30.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c1b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:30.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:30.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c18ff9640 1 --2- 172.21.15.62:0/2255897224 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0bfc036470 0x7f0bfc038930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:30.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c18ff9640 1 -- 172.21.15.62:0/2255897224 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4346+0+0 (secure 0 0 0) 0x7f0c0c0493a0 con 0x7f0c1c0a60b0 2024-04-19T01:42:30.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c1b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:30.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c1b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:30.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c1b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:30.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c2094a640 1 --2- 172.21.15.62:0/2255897224 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0bfc036470 0x7f0bfc038930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:30.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c1b7fe640 10 monclient(hunting): handle_auth_done global_id 4324 payload 824 2024-04-19T01:42:30.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c1b7fe640 1 -- 172.21.15.62:0/2255897224 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0c1c068ab0 con 0x7f0c1c136a00 2024-04-19T01:42:30.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.934+0000 7f0c2094a640 10 monclient: get_auth_request con 0x7f0bfc036470 auth_method 0 2024-04-19T01:42:30.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.938+0000 7f0c1b7fe640 1 -- 172.21.15.62:0/2255897224 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c1c136a00 msgr2=0x7f0c1c1cf470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:30.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.938+0000 7f0c1b7fe640 1 --2- 172.21.15.62:0/2255897224 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c1c136a00 0x7f0c1c1cf470 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c0a2ab0 tx=0x7f0c0400f910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:30.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.938+0000 7f0c18ff9640 1 -- 172.21.15.62:0/2255897224 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c04013a20 con 0x7f0c1c136a00 2024-04-19T01:42:30.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.938+0000 7f0c18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:30.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.938+0000 7f0c18ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:42:30.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.938+0000 7f0c18ff9640 10 monclient: dump: 2024-04-19T01:42:30.947 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:30.947 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:30.947 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.947 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.948 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:30.948 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:30.948 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:30.948 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:30.948 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:30.948 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:30.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.938+0000 7f0c2094a640 1 --2- 172.21.15.62:0/2255897224 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0bfc036470 0x7f0bfc038930 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0c140ac000 tx=0x7f0c140a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:30.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.938+0000 7f0c18ff9640 1 -- 172.21.15.62:0/2255897224 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0c0401a070 con 0x7f0c1c136a00 2024-04-19T01:42:30.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.938+0000 7f0c18ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:30.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.938+0000 7f0c18ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:30.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.938+0000 7f0c18ff9640 1 -- 172.21.15.62:0/2255897224 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c1c136a00 msgr2=0x7f0c1c1cf470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:30.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.938+0000 7f0c18ff9640 1 --2- 172.21.15.62:0/2255897224 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c1c136a00 0x7f0c1c1cf470 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c0a2ab0 tx=0x7f0c0400f910 comp rx=0 tx=0).stop 2024-04-19T01:42:30.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.938+0000 7f0c18ff9640 1 -- 172.21.15.62:0/2255897224 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c1c136a00 msgr2=0x7f0c1c1cf470 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:42:30.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.938+0000 7f0c18ff9640 1 --2- 172.21.15.62:0/2255897224 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c1c136a00 0x7f0c1c1cf470 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c0a2ab0 tx=0x7f0c0400f910 comp rx=0 tx=0).stop 2024-04-19T01:42:30.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.942+0000 7f0c223ad640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:42:30.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.942+0000 7f0c223ad640 1 --2- 172.21.15.62:0/2255897224 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c1c1cf9b0 0x7f0c1c1cfdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:30.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.942+0000 7f0c223ad640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:30.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.942+0000 7f0c223ad640 1 -- 172.21.15.62:0/2255897224 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1c1d0300 con 0x7f0c1c1cf9b0 2024-04-19T01:42:30.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.942+0000 7f0c1b7fe640 1 --2- 172.21.15.62:0/2255897224 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c1c1cf9b0 0x7f0c1c1cfdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:30.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.942+0000 7f0c1b7fe640 10 monclient: get_auth_request con 0x7f0c1c1cf9b0 auth_method 0 2024-04-19T01:42:30.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.942+0000 7f0c1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:30.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.942+0000 7f0c1b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:30.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.942+0000 7f0c1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:30.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.946+0000 7f0c1b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:30.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.946+0000 7f0c1b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:30.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.946+0000 7f0c1b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:30.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.946+0000 7f0c1b7fe640 10 monclient(hunting): handle_auth_done global_id 4327 payload 824 2024-04-19T01:42:30.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.946+0000 7f0c1b7fe640 1 -- 172.21.15.62:0/2255897224 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0c1c1219c0 con 0x7f0c1c1cf9b0 2024-04-19T01:42:30.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.946+0000 7f0c1b7fe640 1 -- 172.21.15.62:0/2255897224 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c1c1cf9b0 msgr2=0x7f0c1c1cfdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:30.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.946+0000 7f0c1b7fe640 1 --2- 172.21.15.62:0/2255897224 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c1c1cf9b0 0x7f0c1c1cfdc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0c04019000 tx=0x7f0c040046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:30.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.946+0000 7f0c18ff9640 1 -- 172.21.15.62:0/2255897224 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c04015820 con 0x7f0c1c1cf9b0 2024-04-19T01:42:30.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.946+0000 7f0c18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:30.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.946+0000 7f0c18ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:42:30.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.946+0000 7f0c18ff9640 10 monclient: dump: 2024-04-19T01:42:30.952 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:30.952 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:30.952 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.952 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:30.952 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:30.952 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:30.952 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:30.952 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:30.952 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:30.952 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:30.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.946+0000 7f0c18ff9640 1 -- 172.21.15.62:0/2255897224 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4716 (secure 0 0 0) 0x7f0c04015820 con 0x7f0c1c1cf9b0 2024-04-19T01:42:30.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.946+0000 7f0c18ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:42:30.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.946+0000 7f0c18ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:30.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.946+0000 7f0c18ff9640 1 -- 172.21.15.62:0/2255897224 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c1c1cf9b0 msgr2=0x7f0c1c1cfdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:30.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.946+0000 7f0c18ff9640 1 --2- 172.21.15.62:0/2255897224 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c1c1cf9b0 0x7f0c1c1cfdc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0c04019000 tx=0x7f0c040046d0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.946+0000 7f0c18ff9640 1 -- 172.21.15.62:0/2255897224 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c1c1cf9b0 msgr2=0x7f0c1c1cfdc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:42:30.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.946+0000 7f0c18ff9640 1 --2- 172.21.15.62:0/2255897224 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c1c1cf9b0 0x7f0c1c1cfdc0 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0c04019000 tx=0x7f0c040046d0 comp rx=0 tx=0).stop 2024-04-19T01:42:30.953 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-19T01:42:30.953 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "b", 2024-04-19T01:42:30.953 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 1, 2024-04-19T01:42:30.953 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "peon", 2024-04-19T01:42:30.954 INFO:teuthology.orchestra.run.smithi062.stdout: "election_epoch": 8, 2024-04-19T01:42:30.954 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 27123, 2024-04-19T01:42:30.954 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum": [ 2024-04-19T01:42:30.954 INFO:teuthology.orchestra.run.smithi062.stdout: 0, 2024-04-19T01:42:30.954 INFO:teuthology.orchestra.run.smithi062.stdout: 1, 2024-04-19T01:42:30.954 INFO:teuthology.orchestra.run.smithi062.stdout: 2 2024-04-19T01:42:30.954 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:42:30.954 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_age": 21, 2024-04-19T01:42:30.954 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:42:30.954 INFO:teuthology.orchestra.run.smithi062.stdout: "required_con": "2450521708007866372", 2024-04-19T01:42:30.954 INFO:teuthology.orchestra.run.smithi062.stdout: "required_mon": [ 2024-04-19T01:42:30.954 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:42:30.954 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:42:30.955 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:42:30.955 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:42:30.955 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:42:30.955 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:42:30.955 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:42:30.955 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:42:30.955 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:42:30.955 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:42:30.955 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:42:30.955 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:42:30.955 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:42:30.955 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_mon": [ 2024-04-19T01:42:30.955 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:42:30.955 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:42:30.955 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:42:30.956 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:42:30.956 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:42:30.956 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:42:30.956 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:42:30.956 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:42:30.956 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:42:30.956 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:42:30.956 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:42:30.956 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:42:30.956 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.956 INFO:teuthology.orchestra.run.smithi062.stdout: "outside_quorum": [], 2024-04-19T01:42:30.956 INFO:teuthology.orchestra.run.smithi062.stdout: "extra_probe_peers": [], 2024-04-19T01:42:30.956 INFO:teuthology.orchestra.run.smithi062.stdout: "sync_provider": [], 2024-04-19T01:42:30.956 INFO:teuthology.orchestra.run.smithi062.stdout: "monmap": { 2024-04-19T01:42:30.956 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 1, 2024-04-19T01:42:30.957 INFO:teuthology.orchestra.run.smithi062.stdout: "fsid": "6466e4b6-2520-485f-95b3-9401915fbbad", 2024-04-19T01:42:30.957 INFO:teuthology.orchestra.run.smithi062.stdout: "modified": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:42:30.957 INFO:teuthology.orchestra.run.smithi062.stdout: "created": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:42:30.957 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release": 19, 2024-04-19T01:42:30.957 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release_name": "squid", 2024-04-19T01:42:30.957 INFO:teuthology.orchestra.run.smithi062.stdout: "election_strategy": 1, 2024-04-19T01:42:30.957 INFO:teuthology.orchestra.run.smithi062.stdout: "disallowed_leaders: ": "", 2024-04-19T01:42:30.957 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false, 2024-04-19T01:42:30.957 INFO:teuthology.orchestra.run.smithi062.stdout: "tiebreaker_mon": "", 2024-04-19T01:42:30.957 INFO:teuthology.orchestra.run.smithi062.stdout: "removed_ranks: ": "", 2024-04-19T01:42:30.957 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:42:30.957 INFO:teuthology.orchestra.run.smithi062.stdout: "persistent": [ 2024-04-19T01:42:30.957 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:42:30.957 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:42:30.957 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:42:30.958 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:42:30.958 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:42:30.958 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:42:30.958 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:42:30.958 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:42:30.958 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:42:30.958 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:42:30.958 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:42:30.958 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:42:30.958 INFO:teuthology.orchestra.run.smithi062.stdout: "optional": [] 2024-04-19T01:42:30.958 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.958 INFO:teuthology.orchestra.run.smithi062.stdout: "mons": [ 2024-04-19T01:42:30.958 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.958 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 0, 2024-04-19T01:42:30.958 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "a", 2024-04-19T01:42:30.959 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:42:30.959 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:42:30.959 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.959 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:42:30.959 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3300", 2024-04-19T01:42:30.959 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:42:30.959 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.959 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.959 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:42:30.959 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789", 2024-04-19T01:42:30.959 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:42:30.959 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:30.959 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:42:30.959 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.959 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789/0", 2024-04-19T01:42:30.959 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6789/0", 2024-04-19T01:42:30.960 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:42:30.960 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:42:30.960 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:42:30.960 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.960 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.960 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 1, 2024-04-19T01:42:30.960 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "b", 2024-04-19T01:42:30.960 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:42:30.960 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:42:30.960 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.960 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:42:30.960 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:3300", 2024-04-19T01:42:30.960 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:42:30.960 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.960 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.960 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:42:30.961 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789", 2024-04-19T01:42:30.961 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:42:30.961 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:30.961 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:42:30.961 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.961 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789/0", 2024-04-19T01:42:30.961 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.71:6789/0", 2024-04-19T01:42:30.961 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:42:30.961 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:42:30.961 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:42:30.961 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.961 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.962 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 2, 2024-04-19T01:42:30.962 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "c", 2024-04-19T01:42:30.962 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:42:30.962 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:42:30.962 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.962 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:42:30.962 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3301", 2024-04-19T01:42:30.963 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:42:30.963 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.963 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.963 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:42:30.963 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790", 2024-04-19T01:42:30.963 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:42:30.963 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:30.963 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:42:30.963 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.963 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790/0", 2024-04-19T01:42:30.963 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6790/0", 2024-04-19T01:42:30.963 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:42:30.963 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:42:30.964 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:42:30.964 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:30.964 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:42:30.964 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.964 INFO:teuthology.orchestra.run.smithi062.stdout: "feature_map": { 2024-04-19T01:42:30.964 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": [ 2024-04-19T01:42:30.964 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.964 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:42:30.964 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:42:30.964 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:42:30.964 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:30.964 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:42:30.964 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": [ 2024-04-19T01:42:30.964 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.965 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:42:30.965 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:42:30.965 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 3 2024-04-19T01:42:30.965 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:30.965 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:42:30.965 INFO:teuthology.orchestra.run.smithi062.stdout: "client": [ 2024-04-19T01:42:30.965 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:30.965 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:42:30.965 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:42:30.965 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 6 2024-04-19T01:42:30.965 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:30.965 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:42:30.965 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:30.965 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false 2024-04-19T01:42:30.965 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-19T01:42:30.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.950+0000 7f0c223ad640 1 -- 172.21.15.62:0/2255897224 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0bfc036470 msgr2=0x7f0bfc038930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:30.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.950+0000 7f0c223ad640 1 --2- 172.21.15.62:0/2255897224 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0bfc036470 0x7f0bfc038930 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0c140ac000 tx=0x7f0c140a7070 comp rx=0 tx=0).stop 2024-04-19T01:42:30.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.950+0000 7f0c223ad640 10 monclient: shutdown 2024-04-19T01:42:30.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.950+0000 7f0c223ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:30.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.950+0000 7f0c223ad640 1 -- 172.21.15.62:0/2255897224 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c1c0a60b0 msgr2=0x7f0c1c120660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:30.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.950+0000 7f0c223ad640 1 --2- 172.21.15.62:0/2255897224 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c1c0a60b0 0x7f0c1c120660 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c009fd0 tx=0x7f0c0c00ee10 comp rx=0 tx=0).stop 2024-04-19T01:42:30.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.950+0000 7f0c2094a640 1 -- 172.21.15.62:0/2255897224 reap_dead start 2024-04-19T01:42:30.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.950+0000 7f0c2094a640 1 -- 172.21.15.62:0/2255897224 reap_dead start 2024-04-19T01:42:30.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.950+0000 7f0c223ad640 1 -- 172.21.15.62:0/2255897224 shutdown_connections 2024-04-19T01:42:30.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.950+0000 7f0c223ad640 1 -- 172.21.15.62:0/2255897224 >> 172.21.15.62:0/2255897224 conn(0x7f0c1c0aa1d0 msgr2=0x7f0c1c0a8260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:30.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.950+0000 7f0c223ad640 1 -- 172.21.15.62:0/2255897224 shutdown_connections 2024-04-19T01:42:30.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:30.950+0000 7f0c223ad640 1 -- 172.21.15.62:0/2255897224 wait complete. 2024-04-19T01:42:30.968 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-19T01:42:30.968 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-04-19T01:42:30.969 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-19T01:42:30.969 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-04-19T01:42:30.969 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-19T01:42:30.969 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-19T01:42:30.969 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:42:30.980 INFO:tasks.ceph.mon.a:Stopped 2024-04-19T01:42:30.981 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-19T01:42:30.981 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:42:31.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234ee4d640 10 monclient: get_monmap_and_config 2024-04-19T01:42:31.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234ee4d640 10 monclient: build_initial_monmap 2024-04-19T01:42:31.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234ee4d640 10 monclient: monmap: 2024-04-19T01:42:31.135 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:31.135 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:31.135 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:31.139290+0000 2024-04-19T01:42:31.135 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:31.139290+0000 2024-04-19T01:42:31.135 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:31.135 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:31.135 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:31.135 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:31.135 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:31.136 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:31.136 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:31.136 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:31.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234ee4d640 1 Processor -- start 2024-04-19T01:42:31.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234ee4d640 1 -- start start 2024-04-19T01:42:31.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234ee4d640 10 monclient: init 2024-04-19T01:42:31.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234ee4d640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:31.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234ee4d640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-04-19T01:42:31.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234ee4d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23480a47e0 0x7f23480a4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:31.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234ee4d640 10 monclient(hunting): picked mon.noname-b con 0x7f23480a47e0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:31.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234ee4d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f23480a5100 addr v1:172.21.15.71:6789/0 2024-04-19T01:42:31.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234ee4d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f23481357c0 0x7f234813d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:31.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234ee4d640 10 monclient(hunting): picked mon.noname-c con 0x7f23481357c0 addr v2:172.21.15.71:3300/0 2024-04-19T01:42:31.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234ee4d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f234809f7c0 con 0x7f23480a5100 2024-04-19T01:42:31.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234ee4d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:31.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234ee4d640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f234809bf00 con 0x7f23481357c0 2024-04-19T01:42:31.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234ee4d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:31.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234ee4d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f234809ee90 con 0x7f23480a47e0 2024-04-19T01:42:31.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234ee4d640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:31.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234ee4d640 10 monclient(hunting): authenticate will time out at 936.958766s 2024-04-19T01:42:31.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234cbe9640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23480a47e0 0x7f23480a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:31.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234cbe9640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23480a47e0 0x7f23480a4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60674/0 (socket says 172.21.15.62:60674) 2024-04-19T01:42:31.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234cbe9640 1 -- 172.21.15.62:0/893959818 learned_addr learned my addr 172.21.15.62:0/893959818 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:31.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234cbe9640 10 monclient(hunting): get_auth_request con 0x7f23480a47e0 auth_method 0 2024-04-19T01:42:31.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234cbe9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:31.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234cbe9640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:31.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234cbe9640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:31.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234cbe9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:31.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234cbe9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:31.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234cbe9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:31.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234d3ea640 1 --2- 172.21.15.62:0/893959818 >> v2:172.21.15.71:3300/0 conn(0x7f23481357c0 0x7f234813d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:31.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234cbe9640 10 monclient(hunting): handle_auth_done global_id 4367 payload 274 2024-04-19T01:42:31.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234cbe9640 1 -- 172.21.15.62:0/893959818 >> v2:172.21.15.71:3300/0 conn(0x7f23481357c0 msgr2=0x7f234813d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:31.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234cbe9640 1 --2- 172.21.15.62:0/893959818 >> v2:172.21.15.71:3300/0 conn(0x7f23481357c0 0x7f234813d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:31.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234cbe9640 1 -- 172.21.15.62:0/893959818 >> v1:172.21.15.71:6789/0 conn(0x7f23480a5100 legacy=0x7f23481350b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:31.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234cbe9640 10 monclient: _finish_hunting 0 2024-04-19T01:42:31.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234cbe9640 1 monclient: found mon.noname-b 2024-04-19T01:42:31.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234cbe9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:31.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234cbe9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:42:31.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.134+0000 7f234cbe9640 1 -- 172.21.15.62:0/893959818 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f234809c2d0 con 0x7f23480a47e0 2024-04-19T01:42:31.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.138+0000 7f234cbe9640 1 -- 172.21.15.62:0/893959818 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23480a47e0 msgr2=0x7f23480a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:31.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.138+0000 7f234cbe9640 1 --2- 172.21.15.62:0/893959818 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23480a47e0 0x7f23480a4bc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f233c009870 tx=0x7f233c01c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0b0dd00f43adc41 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:31.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.138+0000 7f23477fe640 1 -- 172.21.15.62:0/893959818 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f233c0b0070 con 0x7f23480a47e0 2024-04-19T01:42:31.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.138+0000 7f23477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:31.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.138+0000 7f23477fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:42:31.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.138+0000 7f23477fe640 10 monclient: dump: 2024-04-19T01:42:31.147 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:31.147 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:31.147 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:31.147 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:31.147 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:31.148 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:31.148 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:31.148 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:31.148 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:31.148 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:31.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.138+0000 7f23477fe640 10 monclient: _finish_auth 0 2024-04-19T01:42:31.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.138+0000 7f23477fe640 10 monclient: _check_auth_tickets 2024-04-19T01:42:31.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.138+0000 7f23477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:31.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.138+0000 7f234ee4d640 5 monclient: authenticate success, global_id 4367 2024-04-19T01:42:31.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.138+0000 7f234ee4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:42:31.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.138+0000 7f23477fe640 1 -- 172.21.15.62:0/893959818 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f233c0ab600 con 0x7f23480a47e0 2024-04-19T01:42:31.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.138+0000 7f23477fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:31.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.138+0000 7f234ee4d640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:31.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.138+0000 7f23477fe640 1 -- 172.21.15.62:0/893959818 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f233c0ab9d0 con 0x7f23480a47e0 2024-04-19T01:42:31.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.138+0000 7f23477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:31.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.138+0000 7f234ee4d640 4 set_mon_vals no callback set 2024-04-19T01:42:31.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.138+0000 7f23477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:31.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.138+0000 7f23477fe640 10 monclient: dump: 2024-04-19T01:42:31.149 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:31.149 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:31.149 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:31.149 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:31.149 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:31.149 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:31.149 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:31.150 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:31.150 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:31.150 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:31.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234ee4d640 10 monclient: shutdown 2024-04-19T01:42:31.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234ee4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:31.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234ee4d640 1 -- 172.21.15.62:0/893959818 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23480a47e0 msgr2=0x7f23480a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:31.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234ee4d640 1 --2- 172.21.15.62:0/893959818 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23480a47e0 0x7f23480a4bc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f233c009870 tx=0x7f233c01c2d0 comp rx=0 tx=0).stop 2024-04-19T01:42:31.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234ee4d640 1 -- 172.21.15.62:0/893959818 shutdown_connections 2024-04-19T01:42:31.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234ee4d640 1 --2- 172.21.15.62:0/893959818 >> v2:172.21.15.71:3300/0 conn(0x7f23481357c0 0x7f234813d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:31.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234ee4d640 1 --2- 172.21.15.62:0/893959818 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23480a47e0 0x7f23480a4bc0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:31.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234ee4d640 1 -- 172.21.15.62:0/893959818 >> 172.21.15.62:0/893959818 conn(0x7f23480a23d0 msgr2=0x7f23480a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:31.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234ee4d640 1 -- 172.21.15.62:0/893959818 shutdown_connections 2024-04-19T01:42:31.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234ee4d640 1 -- 172.21.15.62:0/893959818 wait complete. 2024-04-19T01:42:31.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234ee4d640 10 monclient: build_initial_monmap 2024-04-19T01:42:31.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234ee4d640 10 monclient: monmap: 2024-04-19T01:42:31.151 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:31.151 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:31.151 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:31.151 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:31.151 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:31.151 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:31.151 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:31.151 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:31.151 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:31.151 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:31.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234ee4d640 1 Processor -- start 2024-04-19T01:42:31.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234ee4d640 1 -- start start 2024-04-19T01:42:31.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234ee4d640 10 monclient: init 2024-04-19T01:42:31.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234ee4d640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:31.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234ee4d640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-19T01:42:31.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234ee4d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f23480a47e0 0x7f23481cf680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:31.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234ee4d640 10 monclient(hunting): picked mon.noname-b con 0x7f23480a47e0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:31.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234ee4d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23480a5100 0x7f2348130260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:31.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234ee4d640 10 monclient(hunting): picked mon.noname-c con 0x7f23480a5100 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:31.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234ee4d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23481357c0 0x7f23481307a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:31.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234ee4d640 10 monclient(hunting): picked mon.noname-a con 0x7f23481357c0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:31.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234ee4d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:31.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234ee4d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f234809c8e0 con 0x7f23481357c0 2024-04-19T01:42:31.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234ee4d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:31.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234ee4d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f234809c760 con 0x7f23480a47e0 2024-04-19T01:42:31.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234ee4d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:31.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234ee4d640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f2348140360 con 0x7f23480a5100 2024-04-19T01:42:31.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234ee4d640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:31.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234ee4d640 10 monclient(hunting): authenticate will time out at 936.966105s 2024-04-19T01:42:31.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234d3ea640 1 -- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23481357c0 msgr2=0x7f23481307a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:3300/0 2024-04-19T01:42:31.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234d3ea640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23481357c0 0x7f23481307a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:42:31.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f2347fff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23480a5100 0x7f2348130260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:31.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f2347fff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23480a5100 0x7f2348130260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60680/0 (socket says 172.21.15.62:60680) 2024-04-19T01:42:31.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f2347fff640 1 -- 172.21.15.62:0/570528887 learned_addr learned my addr 172.21.15.62:0/570528887 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:31.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f2347fff640 10 monclient(hunting): get_auth_request con 0x7f23480a5100 auth_method 0 2024-04-19T01:42:31.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f2347fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:31.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f2347fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:31.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f2347fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:31.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f2347fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:31.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f2347fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:31.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f2347fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:31.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234cbe9640 1 --2- 172.21.15.62:0/570528887 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f23480a47e0 0x7f23481cf680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:31.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234cbe9640 10 monclient(hunting): get_auth_request con 0x7f23480a47e0 auth_method 0 2024-04-19T01:42:31.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234cbe9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:31.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234cbe9640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:31.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f234cbe9640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:31.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f2347fff640 10 monclient(hunting): handle_auth_done global_id 4370 payload 824 2024-04-19T01:42:31.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f2347fff640 1 -- 172.21.15.62:0/570528887 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f23480a47e0 msgr2=0x7f23481cf680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:31.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f2347fff640 1 --2- 172.21.15.62:0/570528887 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f23480a47e0 0x7f23481cf680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:31.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f2347fff640 1 -- 172.21.15.62:0/570528887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23481357c0 msgr2=0x7f23481307a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:42:31.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f2347fff640 1 --2- 172.21.15.62:0/570528887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23481357c0 0x7f23481307a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:31.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f2347fff640 10 monclient: _finish_hunting 0 2024-04-19T01:42:31.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f2347fff640 1 monclient: found mon.noname-c 2024-04-19T01:42:31.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f2347fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:31.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f2347fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:42:31.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.142+0000 7f2347fff640 1 -- 172.21.15.62:0/570528887 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f234811c650 con 0x7f23480a5100 2024-04-19T01:42:31.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.146+0000 7f2347fff640 1 -- 172.21.15.62:0/570528887 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23480a5100 msgr2=0x7f2348130260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:31.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.146+0000 7f2347fff640 1 --2- 172.21.15.62:0/570528887 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23480a5100 0x7f2348130260 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f233800c7c0 tx=0x7f233800cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:31.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.146+0000 7f23457fa640 1 -- 172.21.15.62:0/570528887 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2338016020 con 0x7f23480a5100 2024-04-19T01:42:31.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.146+0000 7f23457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:31.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.146+0000 7f23457fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:31.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.146+0000 7f23457fa640 10 monclient: dump: 2024-04-19T01:42:31.156 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:31.156 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:31.156 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:31.156 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:31.156 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:31.156 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:31.156 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:31.156 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:31.156 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:31.156 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:31.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.146+0000 7f23457fa640 10 monclient: _finish_auth 0 2024-04-19T01:42:31.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.146+0000 7f23457fa640 10 monclient: _check_auth_tickets 2024-04-19T01:42:31.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.146+0000 7f23457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:31.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.146+0000 7f23457fa640 1 -- 172.21.15.62:0/570528887 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f233800ac60 con 0x7f23480a5100 2024-04-19T01:42:31.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.146+0000 7f23457fa640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:31.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.146+0000 7f23457fa640 1 -- 172.21.15.62:0/570528887 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2338012660 con 0x7f23480a5100 2024-04-19T01:42:31.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.146+0000 7f234ee4d640 5 monclient: authenticate success, global_id 4370 2024-04-19T01:42:31.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.146+0000 7f234ee4d640 10 monclient: _renew_subs 2024-04-19T01:42:31.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.146+0000 7f234ee4d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:31.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.146+0000 7f234ee4d640 1 -- 172.21.15.62:0/570528887 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23480707e0 con 0x7f23480a5100 2024-04-19T01:42:31.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.146+0000 7f2346ffd640 4 set_mon_vals no callback set 2024-04-19T01:42:31.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.146+0000 7f23457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:31.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.146+0000 7f23457fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:31.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.146+0000 7f23457fa640 10 monclient: dump: 2024-04-19T01:42:31.157 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:31.158 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:31.158 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:31.158 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:31.158 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:31.158 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:31.158 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:31.158 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:31.158 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:31.158 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:31.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.146+0000 7f234ee4d640 10 monclient: _renew_subs 2024-04-19T01:42:31.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.146+0000 7f234ee4d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:31.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.146+0000 7f234ee4d640 1 -- 172.21.15.62:0/570528887 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f234811bfb0 con 0x7f23480a5100 2024-04-19T01:42:31.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.146+0000 7f234ee4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:31.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.146+0000 7f234ee4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:31.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.146+0000 7f234ee4d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:31.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.146+0000 7f234ee4d640 1 -- 172.21.15.62:0/570528887 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23480692b0 con 0x7f23480a5100 2024-04-19T01:42:31.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.146+0000 7f23457fa640 1 -- 172.21.15.62:0/570528887 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f2338019020 con 0x7f23480a5100 2024-04-19T01:42:31.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.146+0000 7f23457fa640 1 --2- 172.21.15.62:0/570528887 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f23340363d0 0x7f2334038890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:31.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.146+0000 7f23457fa640 1 -- 172.21.15.62:0/570528887 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4346+0+0 (secure 0 0 0) 0x7f2338049420 con 0x7f23480a5100 2024-04-19T01:42:31.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.146+0000 7f234cbe9640 1 --2- 172.21.15.62:0/570528887 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f23340363d0 0x7f2334038890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:31.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.146+0000 7f234cbe9640 10 monclient: get_auth_request con 0x7f23340363d0 auth_method 0 2024-04-19T01:42:31.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.150+0000 7f234cbe9640 1 --2- 172.21.15.62:0/570528887 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f23340363d0 0x7f2334038890 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f234806cc30 tx=0x7f233c0a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:31.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.150+0000 7f23457fa640 1 -- 172.21.15.62:0/570528887 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2338020030 con 0x7f23480a5100 2024-04-19T01:42:31.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.150+0000 7f23457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:31.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.150+0000 7f23457fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:31.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.374+0000 7f234ee4d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:42:31.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.374+0000 7f234ee4d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:42:31.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.374+0000 7f234ee4d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:31.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.374+0000 7f234ee4d640 1 -- 172.21.15.62:0/570528887 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f234806b980 con 0x7f23480a5100 2024-04-19T01:42:31.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.374+0000 7f23457fa640 1 -- 172.21.15.62:0/570528887 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f234811c650 con 0x7f23480a5100 2024-04-19T01:42:31.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.374+0000 7f23457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:42:31.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.374+0000 7f23457fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:31.376 INFO:teuthology.orchestra.run.smithi062.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","modified":"2024-04-19T01:41:38.497795Z","created":"2024-04-19T01:41:38.497795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:42:31.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.378+0000 7f234ee4d640 1 -- 172.21.15.62:0/570528887 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f23340363d0 msgr2=0x7f2334038890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:31.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.378+0000 7f234ee4d640 1 --2- 172.21.15.62:0/570528887 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f23340363d0 0x7f2334038890 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f234806cc30 tx=0x7f233c0a6000 comp rx=0 tx=0).stop 2024-04-19T01:42:31.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.378+0000 7f234ee4d640 10 monclient: shutdown 2024-04-19T01:42:31.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.378+0000 7f234ee4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:31.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.378+0000 7f234ee4d640 1 -- 172.21.15.62:0/570528887 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23480a5100 msgr2=0x7f2348130260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:31.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.378+0000 7f234ee4d640 1 --2- 172.21.15.62:0/570528887 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23480a5100 0x7f2348130260 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f233800c7c0 tx=0x7f233800cc80 comp rx=0 tx=0).stop 2024-04-19T01:42:31.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.378+0000 7f234ee4d640 1 -- 172.21.15.62:0/570528887 shutdown_connections 2024-04-19T01:42:31.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.378+0000 7f234ee4d640 1 --2- 172.21.15.62:0/570528887 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f23340363d0 0x7f2334038890 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:31.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.378+0000 7f234ee4d640 1 --2- 172.21.15.62:0/570528887 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f23480a5100 0x7f2348130260 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:31.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.378+0000 7f234ee4d640 1 --2- 172.21.15.62:0/570528887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23481357c0 0x7f23481307a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:31.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.378+0000 7f234ee4d640 1 --2- 172.21.15.62:0/570528887 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f23480a47e0 0x7f23481cf680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:31.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.378+0000 7f234ee4d640 1 -- 172.21.15.62:0/570528887 >> 172.21.15.62:0/570528887 conn(0x7f23480a23d0 msgr2=0x7f234809d980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:31.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.378+0000 7f234ee4d640 1 -- 172.21.15.62:0/570528887 shutdown_connections 2024-04-19T01:42:31.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:31.378+0000 7f234ee4d640 1 -- 172.21.15.62:0/570528887 wait complete. 2024-04-19T01:42:34.395 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:42:34.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f69334e6640 10 monclient: get_monmap_and_config 2024-04-19T01:42:34.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f69334e6640 10 monclient: build_initial_monmap 2024-04-19T01:42:34.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f69334e6640 10 monclient: monmap: 2024-04-19T01:42:34.528 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:34.528 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:34.528 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:34.531599+0000 2024-04-19T01:42:34.528 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:34.531599+0000 2024-04-19T01:42:34.528 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:34.528 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:34.528 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:34.528 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:34.529 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:34.529 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:34.529 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:34.529 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:34.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f69334e6640 1 Processor -- start 2024-04-19T01:42:34.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f69334e6640 1 -- start start 2024-04-19T01:42:34.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f69334e6640 10 monclient: init 2024-04-19T01:42:34.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f69334e6640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:34.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f69334e6640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-04-19T01:42:34.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f69334e6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f692c0a3410 addr v1:172.21.15.71:6789/0 2024-04-19T01:42:34.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f69334e6640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f692c0a2900 0x7f692c13c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:34.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f69334e6640 10 monclient(hunting): picked mon.noname-a con 0x7f692c0a2900 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:34.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f69334e6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f692c0a2d50 0x7f692c13ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:34.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f69334e6640 10 monclient(hunting): picked mon.noname-b con 0x7f692c0a2d50 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:34.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f69334e6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f692c0a56e0 con 0x7f692c0a3410 2024-04-19T01:42:34.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f69334e6640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:34.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f69334e6640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f692c0a1430 con 0x7f692c0a2900 2024-04-19T01:42:34.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f69334e6640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:34.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f69334e6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f692c0a1710 con 0x7f692c0a2d50 2024-04-19T01:42:34.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f69334e6640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:34.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f69334e6640 10 monclient(hunting): authenticate will time out at 940.351102s 2024-04-19T01:42:34.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f6930a81640 1 -- >> v2:172.21.15.62:3300/0 conn(0x7f692c0a2900 msgr2=0x7f692c13c7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.62:3300/0 2024-04-19T01:42:34.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f6931a83640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f692c0a2d50 0x7f692c13ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:34.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f6930a81640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f692c0a2900 0x7f692c13c7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:42:34.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f6931a83640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f692c0a2d50 0x7f692c13ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60690/0 (socket says 172.21.15.62:60690) 2024-04-19T01:42:34.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f6931a83640 1 -- 172.21.15.62:0/344047073 learned_addr learned my addr 172.21.15.62:0/344047073 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:34.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f6931a83640 10 monclient(hunting): get_auth_request con 0x7f692c0a2d50 auth_method 0 2024-04-19T01:42:34.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f6931a83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:34.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f6931a83640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:34.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f6931a83640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:34.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f6931a83640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:34.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f6931a83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:34.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f6931a83640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:34.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f6931a83640 10 monclient(hunting): handle_auth_done global_id 4373 payload 274 2024-04-19T01:42:34.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f6931a83640 1 -- 172.21.15.62:0/344047073 >> v2:172.21.15.62:3300/0 conn(0x7f692c0a2900 msgr2=0x7f692c13c7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:42:34.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f6931a83640 1 --2- 172.21.15.62:0/344047073 >> v2:172.21.15.62:3300/0 conn(0x7f692c0a2900 0x7f692c13c7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:34.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f6931a83640 1 -- 172.21.15.62:0/344047073 >> v1:172.21.15.71:6789/0 conn(0x7f692c0a3410 legacy=0x7f692c0a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:34.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f6931a83640 10 monclient: _finish_hunting 0 2024-04-19T01:42:34.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f6931a83640 1 monclient: found mon.noname-b 2024-04-19T01:42:34.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f6931a83640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:34.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f6931a83640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:42:34.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.526+0000 7f6931a83640 1 -- 172.21.15.62:0/344047073 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f692c0a19f0 con 0x7f692c0a2d50 2024-04-19T01:42:34.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f6931a83640 1 -- 172.21.15.62:0/344047073 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f692c0a2d50 msgr2=0x7f692c13ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:34.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f6931a83640 1 --2- 172.21.15.62:0/344047073 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f692c0a2d50 0x7f692c13ed00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f692c0a10a0 tx=0x7f692401c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b2f5edd993f70e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:34.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f691ffff640 1 -- 172.21.15.62:0/344047073 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69240b1070 con 0x7f692c0a2d50 2024-04-19T01:42:34.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f691ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:34.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f691ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:42:34.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f691ffff640 10 monclient: dump: 2024-04-19T01:42:34.538 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:34.538 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:34.538 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:34.538 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:34.538 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:34.538 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:34.539 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:34.539 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:34.539 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:34.539 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:34.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f691ffff640 10 monclient: _finish_auth 0 2024-04-19T01:42:34.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f691ffff640 10 monclient: _check_auth_tickets 2024-04-19T01:42:34.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f691ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:34.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 5 monclient: authenticate success, global_id 4373 2024-04-19T01:42:34.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:42:34.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f691ffff640 1 -- 172.21.15.62:0/344047073 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69240ac600 con 0x7f692c0a2d50 2024-04-19T01:42:34.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f691ffff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:34.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:34.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f691ffff640 1 -- 172.21.15.62:0/344047073 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69240ac9d0 con 0x7f692c0a2d50 2024-04-19T01:42:34.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f691ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:34.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 4 set_mon_vals no callback set 2024-04-19T01:42:34.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f691ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:34.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f691ffff640 10 monclient: dump: 2024-04-19T01:42:34.540 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:34.540 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:34.540 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:34.540 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:34.540 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:34.540 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:34.540 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:34.540 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:34.540 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:34.541 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:34.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 10 monclient: shutdown 2024-04-19T01:42:34.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:34.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 1 -- 172.21.15.62:0/344047073 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f692c0a2d50 msgr2=0x7f692c13ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:34.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 1 --2- 172.21.15.62:0/344047073 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f692c0a2d50 0x7f692c13ed00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f692c0a10a0 tx=0x7f692401c400 comp rx=0 tx=0).stop 2024-04-19T01:42:34.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 1 -- 172.21.15.62:0/344047073 shutdown_connections 2024-04-19T01:42:34.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 1 --2- 172.21.15.62:0/344047073 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f692c0a2d50 0x7f692c13ed00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:34.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 1 --2- 172.21.15.62:0/344047073 >> v2:172.21.15.62:3300/0 conn(0x7f692c0a2900 0x7f692c13c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:34.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 1 -- 172.21.15.62:0/344047073 >> 172.21.15.62:0/344047073 conn(0x7f692c0aa2d0 msgr2=0x7f692c0a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:34.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 1 -- 172.21.15.62:0/344047073 shutdown_connections 2024-04-19T01:42:34.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 1 -- 172.21.15.62:0/344047073 wait complete. 2024-04-19T01:42:34.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 10 monclient: build_initial_monmap 2024-04-19T01:42:34.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 10 monclient: monmap: 2024-04-19T01:42:34.542 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:34.542 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:34.542 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:34.542 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:34.542 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:34.542 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:34.542 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:34.542 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:34.542 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:34.542 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:34.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 1 Processor -- start 2024-04-19T01:42:34.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 1 -- start start 2024-04-19T01:42:34.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 10 monclient: init 2024-04-19T01:42:34.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:34.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-19T01:42:34.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f692c0a2900 0x7f692c1cf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:34.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 10 monclient(hunting): picked mon.noname-a con 0x7f692c0a2900 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:34.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f692c0a2d50 0x7f692c123fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:34.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 10 monclient(hunting): picked mon.noname-b con 0x7f692c0a2d50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:34.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f692c0a3410 0x7f692c1244f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:34.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 10 monclient(hunting): picked mon.noname-c con 0x7f692c0a3410 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:34.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:34.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f692c141df0 con 0x7f692c0a2900 2024-04-19T01:42:34.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:34.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f692c141c70 con 0x7f692c0a2d50 2024-04-19T01:42:34.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:34.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f692c141f70 con 0x7f692c0a3410 2024-04-19T01:42:34.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:34.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f69334e6640 10 monclient(hunting): authenticate will time out at 940.358420s 2024-04-19T01:42:34.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f6931282640 1 -- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f692c0a2900 msgr2=0x7f692c1cf8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:3300/0 2024-04-19T01:42:34.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f6931282640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f692c0a2900 0x7f692c1cf8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:42:34.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f6931a83640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f692c0a3410 0x7f692c1244f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:34.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f6931a83640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f692c0a3410 0x7f692c1244f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:60704/0 (socket says 172.21.15.62:60704) 2024-04-19T01:42:34.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f6931a83640 1 -- 172.21.15.62:0/3038502661 learned_addr learned my addr 172.21.15.62:0/3038502661 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:34.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f6931a83640 10 monclient(hunting): get_auth_request con 0x7f692c0a3410 auth_method 0 2024-04-19T01:42:34.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f6931a83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:34.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f6931a83640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:34.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f6931a83640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:34.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f6931a83640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:34.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f6931a83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:34.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f6931a83640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:34.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f6931a83640 10 monclient(hunting): handle_auth_done global_id 4376 payload 824 2024-04-19T01:42:34.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f6931a83640 1 -- 172.21.15.62:0/3038502661 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f692c0a2d50 msgr2=0x7f692c123fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:34.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f6931a83640 1 --2- 172.21.15.62:0/3038502661 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f692c0a2d50 0x7f692c123fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:34.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f6931a83640 1 -- 172.21.15.62:0/3038502661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f692c0a2900 msgr2=0x7f692c1cf8a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:42:34.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f6931a83640 1 --2- 172.21.15.62:0/3038502661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f692c0a2900 0x7f692c1cf8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:34.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f6931a83640 10 monclient: _finish_hunting 0 2024-04-19T01:42:34.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f6931a83640 1 monclient: found mon.noname-c 2024-04-19T01:42:34.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f6931a83640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:34.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f6931a83640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:42:34.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.534+0000 7f6931a83640 1 -- 172.21.15.62:0/3038502661 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f692c13e800 con 0x7f692c0a3410 2024-04-19T01:42:34.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.538+0000 7f6931a83640 1 -- 172.21.15.62:0/3038502661 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f692c0a3410 msgr2=0x7f692c1244f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:34.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.538+0000 7f6931a83640 1 --2- 172.21.15.62:0/3038502661 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f692c0a3410 0x7f692c1244f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f692401cca0 tx=0x7f692401ac40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:34.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.538+0000 7f691dffb640 1 -- 172.21.15.62:0/3038502661 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69240bb020 con 0x7f692c0a3410 2024-04-19T01:42:34.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.538+0000 7f691dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:34.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.538+0000 7f691dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:34.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.538+0000 7f691dffb640 10 monclient: dump: 2024-04-19T01:42:34.548 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:34.548 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:34.548 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:34.548 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:34.549 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:34.549 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:34.549 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:34.549 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:34.549 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:34.549 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:34.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.538+0000 7f691dffb640 10 monclient: _finish_auth 0 2024-04-19T01:42:34.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.538+0000 7f691dffb640 10 monclient: _check_auth_tickets 2024-04-19T01:42:34.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.538+0000 7f691dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:34.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.538+0000 7f69334e6640 5 monclient: authenticate success, global_id 4376 2024-04-19T01:42:34.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.538+0000 7f69334e6640 10 monclient: _renew_subs 2024-04-19T01:42:34.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.538+0000 7f691dffb640 1 -- 172.21.15.62:0/3038502661 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69240b7910 con 0x7f692c0a3410 2024-04-19T01:42:34.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.538+0000 7f69334e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:34.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.538+0000 7f69334e6640 1 -- 172.21.15.62:0/3038502661 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f692c0a3f80 con 0x7f692c0a3410 2024-04-19T01:42:34.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.538+0000 7f691dffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:34.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.538+0000 7f691dffb640 1 -- 172.21.15.62:0/3038502661 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69240b1040 con 0x7f692c0a3410 2024-04-19T01:42:34.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.538+0000 7f691dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:34.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.538+0000 7f691dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:34.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.538+0000 7f691dffb640 10 monclient: dump: 2024-04-19T01:42:34.550 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:34.550 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:34.550 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:34.550 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:34.550 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:34.550 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:34.550 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:34.551 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:34.551 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:34.551 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:34.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.538+0000 7f69334e6640 10 monclient: _renew_subs 2024-04-19T01:42:34.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.538+0000 7f69334e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:34.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.538+0000 7f69334e6640 1 -- 172.21.15.62:0/3038502661 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f692c127c10 con 0x7f692c0a3410 2024-04-19T01:42:34.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.538+0000 7f691f7fe640 4 set_mon_vals no callback set 2024-04-19T01:42:34.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.538+0000 7f691dffb640 1 -- 172.21.15.62:0/3038502661 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f69240be020 con 0x7f692c0a3410 2024-04-19T01:42:34.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.538+0000 7f69334e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:34.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.538+0000 7f69334e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:34.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.538+0000 7f69334e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:34.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.538+0000 7f69334e6640 1 -- 172.21.15.62:0/3038502661 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f692c0692b0 con 0x7f692c0a3410 2024-04-19T01:42:34.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.538+0000 7f691dffb640 1 --2- 172.21.15.62:0/3038502661 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f6910036420 0x7f69100388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:34.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.538+0000 7f691dffb640 1 -- 172.21.15.62:0/3038502661 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4346+0+0 (secure 0 0 0) 0x7f69240e38d0 con 0x7f692c0a3410 2024-04-19T01:42:34.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.538+0000 7f6931282640 1 --2- 172.21.15.62:0/3038502661 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f6910036420 0x7f69100388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:34.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.538+0000 7f6931282640 10 monclient: get_auth_request con 0x7f6910036420 auth_method 0 2024-04-19T01:42:34.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.542+0000 7f6931282640 1 --2- 172.21.15.62:0/3038502661 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f6910036420 0x7f69100388e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6920003780 tx=0x7f6920007a20 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:34.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.542+0000 7f691dffb640 1 -- 172.21.15.62:0/3038502661 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f69240c5030 con 0x7f692c0a3410 2024-04-19T01:42:34.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.542+0000 7f691dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:34.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.542+0000 7f691dffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:34.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.766+0000 7f69334e6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:42:34.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.766+0000 7f69334e6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:42:34.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.766+0000 7f69334e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:34.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.766+0000 7f69334e6640 1 -- 172.21.15.62:0/3038502661 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f692c127fb0 con 0x7f692c0a3410 2024-04-19T01:42:34.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.766+0000 7f691dffb640 1 -- 172.21.15.62:0/3038502661 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f692c13e800 con 0x7f692c0a3410 2024-04-19T01:42:34.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.766+0000 7f691dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:42:34.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.766+0000 7f691dffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:34.767 INFO:teuthology.orchestra.run.smithi062.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","modified":"2024-04-19T01:41:38.497795Z","created":"2024-04-19T01:41:38.497795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:42:34.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.766+0000 7f69334e6640 1 -- 172.21.15.62:0/3038502661 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f6910036420 msgr2=0x7f69100388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:34.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.766+0000 7f69334e6640 1 --2- 172.21.15.62:0/3038502661 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f6910036420 0x7f69100388e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6920003780 tx=0x7f6920007a20 comp rx=0 tx=0).stop 2024-04-19T01:42:34.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.766+0000 7f69334e6640 10 monclient: shutdown 2024-04-19T01:42:34.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.766+0000 7f69334e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:34.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.766+0000 7f69334e6640 1 -- 172.21.15.62:0/3038502661 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f692c0a3410 msgr2=0x7f692c1244f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:34.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.766+0000 7f69334e6640 1 --2- 172.21.15.62:0/3038502661 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f692c0a3410 0x7f692c1244f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f692401cca0 tx=0x7f692401ac40 comp rx=0 tx=0).stop 2024-04-19T01:42:34.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.766+0000 7f69334e6640 1 -- 172.21.15.62:0/3038502661 shutdown_connections 2024-04-19T01:42:34.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.766+0000 7f69334e6640 1 --2- 172.21.15.62:0/3038502661 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f6910036420 0x7f69100388e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:34.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.766+0000 7f69334e6640 1 --2- 172.21.15.62:0/3038502661 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f692c0a3410 0x7f692c1244f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:34.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.766+0000 7f69334e6640 1 --2- 172.21.15.62:0/3038502661 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f692c0a2d50 0x7f692c123fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:34.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.766+0000 7f69334e6640 1 --2- 172.21.15.62:0/3038502661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f692c0a2900 0x7f692c1cf8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:34.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.766+0000 7f69334e6640 1 -- 172.21.15.62:0/3038502661 >> 172.21.15.62:0/3038502661 conn(0x7f692c0aa2d0 msgr2=0x7f692c0a9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:34.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.766+0000 7f69334e6640 1 -- 172.21.15.62:0/3038502661 shutdown_connections 2024-04-19T01:42:34.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:34.766+0000 7f69334e6640 1 -- 172.21.15.62:0/3038502661 wait complete. 2024-04-19T01:42:37.785 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:42:37.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.918+0000 7fdccd208640 10 monclient: get_monmap_and_config 2024-04-19T01:42:37.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.918+0000 7fdccd208640 10 monclient: build_initial_monmap 2024-04-19T01:42:37.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.918+0000 7fdccd208640 10 monclient: monmap: 2024-04-19T01:42:37.919 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:37.919 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:37.919 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:37.923498+0000 2024-04-19T01:42:37.919 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:37.923498+0000 2024-04-19T01:42:37.919 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:37.919 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:37.919 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:37.919 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:37.920 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:37.920 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:37.920 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:37.920 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:37.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.918+0000 7fdccd208640 1 Processor -- start 2024-04-19T01:42:37.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.918+0000 7fdccd208640 1 -- start start 2024-04-19T01:42:37.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.918+0000 7fdccd208640 10 monclient: init 2024-04-19T01:42:37.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.918+0000 7fdccd208640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:37.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.918+0000 7fdccd208640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-04-19T01:42:37.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.918+0000 7fdccd208640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdcc80a47e0 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:37.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.918+0000 7fdccd208640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdcc80a4bc0 addr v1:172.21.15.71:6789/0 2024-04-19T01:42:37.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.918+0000 7fdccd208640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fdcc813a570 0x7fdcc813c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:37.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.918+0000 7fdccd208640 10 monclient(hunting): picked mon.noname-c con 0x7fdcc813a570 addr v2:172.21.15.71:3300/0 2024-04-19T01:42:37.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.918+0000 7fdccd208640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcc809fa60 con 0x7fdcc80a47e0 2024-04-19T01:42:37.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.918+0000 7fdccd208640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcc809ec50 con 0x7fdcc80a4bc0 2024-04-19T01:42:37.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.918+0000 7fdccd208640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:37.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.918+0000 7fdccd208640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fdcc809e690 con 0x7fdcc813a570 2024-04-19T01:42:37.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.918+0000 7fdccd208640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:37.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.918+0000 7fdcc6d9d640 1 -- >> v1:172.21.15.62:6789/0 conn(0x7fdcc80a47e0 legacy=0x7fdcc809f350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.62:6789/0 2024-04-19T01:42:37.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.918+0000 7fdccd208640 10 monclient(hunting): authenticate will time out at 943.743822s 2024-04-19T01:42:37.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.922+0000 7fdcc759e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fdcc813a570 0x7fdcc813c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:37.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.922+0000 7fdcc759e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fdcc813a570 0x7fdcc813c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.62:37308/0 (socket says 172.21.15.62:37308) 2024-04-19T01:42:37.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.922+0000 7fdcc759e640 1 -- 172.21.15.62:0/383723438 learned_addr learned my addr 172.21.15.62:0/383723438 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:37.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.922+0000 7fdcc759e640 10 monclient(hunting): get_auth_request con 0x7fdcc813a570 auth_method 0 2024-04-19T01:42:37.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.922+0000 7fdcc759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:37.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.922+0000 7fdcc759e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:37.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.922+0000 7fdcc759e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:37.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.922+0000 7fdcc759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:37.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.922+0000 7fdcc759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:37.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.922+0000 7fdcc759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:37.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.922+0000 7fdcc759e640 10 monclient(hunting): handle_auth_done global_id 4333 payload 274 2024-04-19T01:42:37.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.922+0000 7fdcc759e640 1 -- 172.21.15.62:0/383723438 >> v1:172.21.15.71:6789/0 conn(0x7fdcc80a4bc0 legacy=0x7fdcc80a5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:37.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.922+0000 7fdcc759e640 1 -- 172.21.15.62:0/383723438 >> v1:172.21.15.62:6789/0 conn(0x7fdcc80a47e0 legacy=0x7fdcc809f350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:42:37.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.922+0000 7fdcc759e640 10 monclient: _finish_hunting 0 2024-04-19T01:42:37.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.922+0000 7fdcc759e640 1 monclient: found mon.noname-c 2024-04-19T01:42:37.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.922+0000 7fdcc759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:37.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.922+0000 7fdcc759e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-04-19T01:42:37.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.922+0000 7fdcc759e640 1 -- 172.21.15.62:0/383723438 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcc809e970 con 0x7fdcc813a570 2024-04-19T01:42:37.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.926+0000 7fdcc759e640 1 -- 172.21.15.62:0/383723438 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcc813a570 msgr2=0x7fdcc813c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:37.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.926+0000 7fdcc759e640 1 --2- 172.21.15.62:0/383723438 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcc813a570 0x7fdcc813c960 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0009e80 tx=0x7fdcc001c960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f37d919fa3d48280 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:37.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.926+0000 7fdcc659c640 1 -- 172.21.15.62:0/383723438 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc00a56b0 con 0x7fdcc813a570 2024-04-19T01:42:37.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.926+0000 7fdcc659c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:37.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.926+0000 7fdcc659c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:37.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.926+0000 7fdcc659c640 10 monclient: dump: 2024-04-19T01:42:37.927 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:37.927 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:37.927 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:37.928 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:37.928 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:37.928 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:37.928 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:37.928 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:37.928 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:37.928 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:37.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.926+0000 7fdcc659c640 10 monclient: _finish_auth 0 2024-04-19T01:42:37.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.926+0000 7fdcc659c640 10 monclient: _check_auth_tickets 2024-04-19T01:42:37.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.926+0000 7fdcc659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:37.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.926+0000 7fdcc659c640 1 -- 172.21.15.62:0/383723438 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcc00a5850 con 0x7fdcc813a570 2024-04-19T01:42:37.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.926+0000 7fdcc659c640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:37.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.926+0000 7fdcc659c640 1 -- 172.21.15.62:0/383723438 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc00a5c20 con 0x7fdcc813a570 2024-04-19T01:42:37.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.926+0000 7fdcc659c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:37.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.926+0000 7fdcc659c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:42:37.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.926+0000 7fdcc659c640 10 monclient: dump: 2024-04-19T01:42:37.929 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:37.929 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:37.929 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:37.929 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:37.929 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:37.929 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:37.929 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:37.929 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:37.929 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:37.929 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:37.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.926+0000 7fdccd208640 5 monclient: authenticate success, global_id 4333 2024-04-19T01:42:37.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.926+0000 7fdccd208640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:37.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.926+0000 7fdccd208640 4 set_mon_vals no callback set 2024-04-19T01:42:37.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.926+0000 7fdccd208640 10 monclient: shutdown 2024-04-19T01:42:37.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.926+0000 7fdccd208640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:37.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.926+0000 7fdccd208640 1 -- 172.21.15.62:0/383723438 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcc813a570 msgr2=0x7fdcc813c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:37.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.926+0000 7fdccd208640 1 --2- 172.21.15.62:0/383723438 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcc813a570 0x7fdcc813c960 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0009e80 tx=0x7fdcc001c960 comp rx=0 tx=0).stop 2024-04-19T01:42:37.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.926+0000 7fdccd208640 1 -- 172.21.15.62:0/383723438 shutdown_connections 2024-04-19T01:42:37.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.926+0000 7fdccd208640 1 --2- 172.21.15.62:0/383723438 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcc813a570 0x7fdcc813c960 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:37.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.926+0000 7fdccd208640 1 -- 172.21.15.62:0/383723438 >> 172.21.15.62:0/383723438 conn(0x7fdcc80a23d0 msgr2=0x7fdcc80a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:37.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.926+0000 7fdccd208640 1 -- 172.21.15.62:0/383723438 shutdown_connections 2024-04-19T01:42:37.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.926+0000 7fdccd208640 1 -- 172.21.15.62:0/383723438 wait complete. 2024-04-19T01:42:37.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.926+0000 7fdccd208640 10 monclient: build_initial_monmap 2024-04-19T01:42:37.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.926+0000 7fdccd208640 10 monclient: monmap: 2024-04-19T01:42:37.933 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:37.933 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:37.933 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:37.933 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:37.933 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:37.933 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:37.933 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:37.933 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:37.933 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:37.933 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:37.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdccd208640 1 Processor -- start 2024-04-19T01:42:37.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdccd208640 1 -- start start 2024-04-19T01:42:37.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdccd208640 10 monclient: init 2024-04-19T01:42:37.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdccd208640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:37.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdccd208640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-19T01:42:37.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdccd208640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdcc80a47e0 0x7fdcc8070780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:37.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdccd208640 10 monclient(hunting): picked mon.noname-a con 0x7fdcc80a47e0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:37.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdccd208640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdcc80a4bc0 0x7fdcc8070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:37.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdccd208640 10 monclient(hunting): picked mon.noname-c con 0x7fdcc80a4bc0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:37.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdccd208640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcc813a570 0x7fdcc8071200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:37.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdccd208640 10 monclient(hunting): picked mon.noname-b con 0x7fdcc813a570 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:37.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdccd208640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:37.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdccd208640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc813fdd0 con 0x7fdcc80a47e0 2024-04-19T01:42:37.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdccd208640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:37.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdcc6d9d640 1 -- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdcc80a47e0 msgr2=0x7fdcc8070780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:3300/0 2024-04-19T01:42:37.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdccd208640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc813fc50 con 0x7fdcc813a570 2024-04-19T01:42:37.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdccd208640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:37.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdccd208640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fdcc813ff50 con 0x7fdcc80a4bc0 2024-04-19T01:42:37.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdccd208640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:37.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdccd208640 10 monclient(hunting): authenticate will time out at 943.752852s 2024-04-19T01:42:37.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdcc6d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdcc80a47e0 0x7fdcc8070780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:42:37.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdcbffff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdcc80a4bc0 0x7fdcc8070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:37.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdcbffff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdcc80a4bc0 0x7fdcc8070cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:50874/0 (socket says 172.21.15.62:50874) 2024-04-19T01:42:37.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdcbffff640 1 -- 172.21.15.62:0/3577466004 learned_addr learned my addr 172.21.15.62:0/3577466004 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:37.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdcbffff640 10 monclient(hunting): get_auth_request con 0x7fdcc80a4bc0 auth_method 0 2024-04-19T01:42:37.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdcbffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:37.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdcbffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:37.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdcbffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:37.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdcbffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:37.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdcbffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:37.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdcbffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:37.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdcbffff640 10 monclient(hunting): handle_auth_done global_id 4379 payload 824 2024-04-19T01:42:37.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdcbffff640 1 -- 172.21.15.62:0/3577466004 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcc813a570 msgr2=0x7fdcc8071200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:37.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdcbffff640 1 --2- 172.21.15.62:0/3577466004 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcc813a570 0x7fdcc8071200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:37.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdcbffff640 1 -- 172.21.15.62:0/3577466004 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdcc80a47e0 msgr2=0x7fdcc8070780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:42:37.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdcbffff640 1 --2- 172.21.15.62:0/3577466004 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdcc80a47e0 0x7fdcc8070780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:37.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdcbffff640 10 monclient: _finish_hunting 0 2024-04-19T01:42:37.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdcbffff640 1 monclient: found mon.noname-c 2024-04-19T01:42:37.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdcbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:37.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdcbffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:42:37.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdcbffff640 1 -- 172.21.15.62:0/3577466004 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcc811f1b0 con 0x7fdcc80a4bc0 2024-04-19T01:42:37.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdcbffff640 1 -- 172.21.15.62:0/3577466004 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdcc80a4bc0 msgr2=0x7fdcc8070cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:37.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.930+0000 7fdcbffff640 1 --2- 172.21.15.62:0/3577466004 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdcc80a4bc0 0x7fdcc8070cc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdcb800c480 tx=0x7fdcb800c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:37.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.934+0000 7fdcbf7fe640 1 -- 172.21.15.62:0/3577466004 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb80119a0 con 0x7fdcc80a4bc0 2024-04-19T01:42:37.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.934+0000 7fdcbf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:37.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.934+0000 7fdcbf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:37.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.934+0000 7fdcbf7fe640 10 monclient: dump: 2024-04-19T01:42:37.938 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:37.939 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:37.939 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:37.939 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:37.939 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:37.939 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:37.939 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:37.939 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:37.939 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:37.939 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:37.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.934+0000 7fdcbf7fe640 10 monclient: _finish_auth 0 2024-04-19T01:42:37.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.934+0000 7fdcbf7fe640 10 monclient: _check_auth_tickets 2024-04-19T01:42:37.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.934+0000 7fdcbf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:37.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.934+0000 7fdcbf7fe640 1 -- 172.21.15.62:0/3577466004 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcb8017070 con 0x7fdcc80a4bc0 2024-04-19T01:42:37.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.934+0000 7fdcbf7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:37.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.934+0000 7fdcbf7fe640 1 -- 172.21.15.62:0/3577466004 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb8011da0 con 0x7fdcc80a4bc0 2024-04-19T01:42:37.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.934+0000 7fdcbf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:37.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.934+0000 7fdcbf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:37.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.934+0000 7fdcbf7fe640 10 monclient: dump: 2024-04-19T01:42:37.940 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:37.940 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:37.940 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:37.940 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:37.940 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:37.940 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:37.940 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:37.940 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:37.940 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:37.940 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:37.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.934+0000 7fdccd208640 5 monclient: authenticate success, global_id 4379 2024-04-19T01:42:37.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.934+0000 7fdccd208640 10 monclient: _renew_subs 2024-04-19T01:42:37.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.934+0000 7fdccd208640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:37.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.934+0000 7fdcc5d9b640 4 set_mon_vals no callback set 2024-04-19T01:42:37.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.934+0000 7fdccd208640 1 -- 172.21.15.62:0/3577466004 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcc809c140 con 0x7fdcc80a4bc0 2024-04-19T01:42:37.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.934+0000 7fdccd208640 10 monclient: _renew_subs 2024-04-19T01:42:37.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.934+0000 7fdccd208640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:37.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.934+0000 7fdccd208640 1 -- 172.21.15.62:0/3577466004 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcc811f700 con 0x7fdcc80a4bc0 2024-04-19T01:42:37.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.934+0000 7fdccd208640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:37.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.934+0000 7fdccd208640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:37.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.934+0000 7fdccd208640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:37.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.934+0000 7fdccd208640 1 -- 172.21.15.62:0/3577466004 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcc80692b0 con 0x7fdcc80a4bc0 2024-04-19T01:42:37.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.934+0000 7fdcbf7fe640 1 -- 172.21.15.62:0/3577466004 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fdcb801c460 con 0x7fdcc80a4bc0 2024-04-19T01:42:37.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.934+0000 7fdcbf7fe640 1 --2- 172.21.15.62:0/3577466004 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fdca80363d0 0x7fdca8038890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:37.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.934+0000 7fdcbf7fe640 1 -- 172.21.15.62:0/3577466004 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4346+0+0 (secure 0 0 0) 0x7fdcb8049280 con 0x7fdcc80a4bc0 2024-04-19T01:42:37.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.934+0000 7fdcc6d9d640 1 --2- 172.21.15.62:0/3577466004 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fdca80363d0 0x7fdca8038890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:37.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.934+0000 7fdcc6d9d640 10 monclient: get_auth_request con 0x7fdca80363d0 auth_method 0 2024-04-19T01:42:37.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.938+0000 7fdcc6d9d640 1 --2- 172.21.15.62:0/3577466004 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fdca80363d0 0x7fdca8038890 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0006fd0 tx=0x7fdcb0006d40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:37.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.938+0000 7fdcbf7fe640 1 -- 172.21.15.62:0/3577466004 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fdcb801a520 con 0x7fdcc80a4bc0 2024-04-19T01:42:37.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.938+0000 7fdcbf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:37.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:37.938+0000 7fdcbf7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:38.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:38.166+0000 7fdccd208640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:42:38.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:38.166+0000 7fdccd208640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:42:38.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:38.166+0000 7fdccd208640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:38.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:38.166+0000 7fdccd208640 1 -- 172.21.15.62:0/3577466004 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fdcc806b980 con 0x7fdcc80a4bc0 2024-04-19T01:42:38.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:38.166+0000 7fdcbf7fe640 1 -- 172.21.15.62:0/3577466004 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fdcb8049080 con 0x7fdcc80a4bc0 2024-04-19T01:42:38.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:38.166+0000 7fdcbf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:42:38.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:38.166+0000 7fdcbf7fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:38.167 INFO:teuthology.orchestra.run.smithi062.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":29,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","modified":"2024-04-19T01:41:38.497795Z","created":"2024-04-19T01:41:38.497795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:42:38.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:38.166+0000 7fdccd208640 1 -- 172.21.15.62:0/3577466004 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fdca80363d0 msgr2=0x7fdca8038890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:38.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:38.166+0000 7fdccd208640 1 --2- 172.21.15.62:0/3577466004 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fdca80363d0 0x7fdca8038890 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0006fd0 tx=0x7fdcb0006d40 comp rx=0 tx=0).stop 2024-04-19T01:42:38.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:38.166+0000 7fdccd208640 10 monclient: shutdown 2024-04-19T01:42:38.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:38.166+0000 7fdccd208640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:38.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:38.166+0000 7fdccd208640 1 -- 172.21.15.62:0/3577466004 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdcc80a4bc0 msgr2=0x7fdcc8070cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:38.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:38.166+0000 7fdccd208640 1 --2- 172.21.15.62:0/3577466004 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdcc80a4bc0 0x7fdcc8070cc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdcb800c480 tx=0x7fdcb800c940 comp rx=0 tx=0).stop 2024-04-19T01:42:38.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:38.170+0000 7fdccd208640 1 -- 172.21.15.62:0/3577466004 shutdown_connections 2024-04-19T01:42:38.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:38.170+0000 7fdccd208640 1 --2- 172.21.15.62:0/3577466004 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fdca80363d0 0x7fdca8038890 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:38.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:38.170+0000 7fdccd208640 1 --2- 172.21.15.62:0/3577466004 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdcc80a4bc0 0x7fdcc8070cc0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:38.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:38.170+0000 7fdccd208640 1 --2- 172.21.15.62:0/3577466004 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdcc813a570 0x7fdcc8071200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:38.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:38.170+0000 7fdccd208640 1 --2- 172.21.15.62:0/3577466004 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdcc80a47e0 0x7fdcc8070780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:38.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:38.170+0000 7fdccd208640 1 -- 172.21.15.62:0/3577466004 >> 172.21.15.62:0/3577466004 conn(0x7fdcc80a23d0 msgr2=0x7fdcc80a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:38.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:38.170+0000 7fdccd208640 1 -- 172.21.15.62:0/3577466004 shutdown_connections 2024-04-19T01:42:38.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:38.170+0000 7fdccd208640 1 -- 172.21.15.62:0/3577466004 wait complete. 2024-04-19T01:42:41.187 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:42:41.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.318+0000 7f74a1539640 10 monclient: get_monmap_and_config 2024-04-19T01:42:41.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.318+0000 7f74a1539640 10 monclient: build_initial_monmap 2024-04-19T01:42:41.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.318+0000 7f74a1539640 10 monclient: monmap: 2024-04-19T01:42:41.320 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:41.320 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:41.320 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:41.324456+0000 2024-04-19T01:42:41.320 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:41.324456+0000 2024-04-19T01:42:41.320 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:41.320 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:41.320 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:41.321 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:41.321 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:41.321 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:41.321 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:41.321 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:41.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.318+0000 7f74a1539640 1 Processor -- start 2024-04-19T01:42:41.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.318+0000 7f74a1539640 1 -- start start 2024-04-19T01:42:41.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.318+0000 7f74a1539640 10 monclient: init 2024-04-19T01:42:41.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.318+0000 7f74a1539640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:41.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.318+0000 7f74a1539640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-04-19T01:42:41.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.318+0000 7f74a1539640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f749c0724f0 0x7f749c0a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:41.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.318+0000 7f74a1539640 10 monclient(hunting): picked mon.noname-c con 0x7f749c0724f0 addr v2:172.21.15.71:3300/0 2024-04-19T01:42:41.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.318+0000 7f74a1539640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f749c0a5090 addr v1:172.21.15.62:6789/0 2024-04-19T01:42:41.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.318+0000 7f74a1539640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f749c137ac0 addr v1:172.21.15.71:6789/0 2024-04-19T01:42:41.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.318+0000 7f74a1539640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f749c0a4480 con 0x7f749c0a5090 2024-04-19T01:42:41.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.318+0000 7f74a1539640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f749c0a4720 con 0x7f749c137ac0 2024-04-19T01:42:41.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.318+0000 7f74a1539640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:41.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.318+0000 7f74a1539640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f749c09c500 con 0x7f749c0724f0 2024-04-19T01:42:41.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.318+0000 7f74a1539640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:41.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.318+0000 7f74a1539640 10 monclient(hunting): authenticate will time out at 947.143897s 2024-04-19T01:42:41.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.318+0000 7f749a7fc640 1 -- >> v1:172.21.15.62:6789/0 conn(0x7f749c0a5090 legacy=0x7f749c1373b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.62:6789/0 2024-04-19T01:42:41.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.322+0000 7f749affd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f749c0724f0 0x7f749c0a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:41.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.322+0000 7f749affd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f749c0724f0 0x7f749c0a4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.62:37330/0 (socket says 172.21.15.62:37330) 2024-04-19T01:42:41.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.322+0000 7f749b7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f749c137ac0 0x7f749c13f6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.62:52372/0 (socket says 172.21.15.62:52372) 2024-04-19T01:42:41.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.322+0000 7f749affd640 1 -- 172.21.15.62:0/2596842009 learned_addr learned my addr 172.21.15.62:0/2596842009 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:41.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.322+0000 7f749affd640 10 monclient(hunting): get_auth_request con 0x7f749c0724f0 auth_method 0 2024-04-19T01:42:41.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.322+0000 7f749affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:41.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.322+0000 7f749affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:41.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.322+0000 7f749affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:41.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.322+0000 7f749affd640 1 -- 172.21.15.62:0/2596842009 >> v2:172.21.15.71:3300/0 conn(0x7f749c0724f0 msgr2=0x7f749c0a4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:42:41.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.322+0000 7f749affd640 1 -- 172.21.15.62:0/2596842009 >> v2:172.21.15.71:3300/0 conn(0x7f749c0724f0 msgr2=0x7f749c0a4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:42:41.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.322+0000 7f749affd640 1 --2- 172.21.15.62:0/2596842009 >> v2:172.21.15.71:3300/0 conn(0x7f749c0724f0 0x7f749c0a4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:42:41.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.322+0000 7f749affd640 1 --2- 172.21.15.62:0/2596842009 >> v2:172.21.15.71:3300/0 conn(0x7f749c0724f0 0x7f749c0a4b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:42:41.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.322+0000 7f749b7fe640 1 -- 172.21.15.62:0/2596842009 >> v1:172.21.15.71:6789/0 conn(0x7f749c137ac0 legacy=0x7f749c13f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:41.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.526+0000 7f749a7fc640 1 -- 172.21.15.62:0/2596842009 >> v1:172.21.15.62:6789/0 conn(0x7f749c0a5090 legacy=0x7f749c1373b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.62:6789/0 2024-04-19T01:42:41.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.526+0000 7f749affd640 1 --2- 172.21.15.62:0/2596842009 >> v2:172.21.15.71:3300/0 conn(0x7f749c0724f0 0x7f749c0a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:41.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.526+0000 7f749affd640 10 monclient(hunting): get_auth_request con 0x7f749c0724f0 auth_method 0 2024-04-19T01:42:41.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.526+0000 7f749affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:41.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.526+0000 7f749affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:41.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.526+0000 7f749affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:42:41.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.526+0000 7f749affd640 1 -- 172.21.15.62:0/2596842009 >> v2:172.21.15.71:3300/0 conn(0x7f749c0724f0 msgr2=0x7f749c0a4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:42:41.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.526+0000 7f749affd640 1 -- 172.21.15.62:0/2596842009 >> v2:172.21.15.71:3300/0 conn(0x7f749c0724f0 msgr2=0x7f749c0a4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:42:41.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.526+0000 7f749affd640 1 --2- 172.21.15.62:0/2596842009 >> v2:172.21.15.71:3300/0 conn(0x7f749c0724f0 0x7f749c0a4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:42:41.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.526+0000 7f749affd640 1 --2- 172.21.15.62:0/2596842009 >> v2:172.21.15.71:3300/0 conn(0x7f749c0724f0 0x7f749c0a4b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:42:41.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.930+0000 7f749a7fc640 1 -- 172.21.15.62:0/2596842009 >> v1:172.21.15.62:6789/0 conn(0x7f749c0a5090 legacy=0x7f749c1373b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.62:6789/0 2024-04-19T01:42:41.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.930+0000 7f749affd640 1 --2- 172.21.15.62:0/2596842009 >> v2:172.21.15.71:3300/0 conn(0x7f749c0724f0 0x7f749c0a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:41.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.930+0000 7f749affd640 10 monclient(hunting): get_auth_request con 0x7f749c0724f0 auth_method 0 2024-04-19T01:42:41.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.930+0000 7f749affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:41.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.930+0000 7f749affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:41.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.930+0000 7f749affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:42:41.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.930+0000 7f749affd640 1 -- 172.21.15.62:0/2596842009 >> v2:172.21.15.71:3300/0 conn(0x7f749c0724f0 msgr2=0x7f749c0a4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:42:41.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.930+0000 7f749affd640 1 -- 172.21.15.62:0/2596842009 >> v2:172.21.15.71:3300/0 conn(0x7f749c0724f0 msgr2=0x7f749c0a4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:42:41.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.930+0000 7f749affd640 1 --2- 172.21.15.62:0/2596842009 >> v2:172.21.15.71:3300/0 conn(0x7f749c0724f0 0x7f749c0a4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:42:41.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:41.930+0000 7f749affd640 1 --2- 172.21.15.62:0/2596842009 >> v2:172.21.15.71:3300/0 conn(0x7f749c0724f0 0x7f749c0a4b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:42:42.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:42.734+0000 7f749a7fc640 1 -- 172.21.15.62:0/2596842009 >> v1:172.21.15.62:6789/0 conn(0x7f749c0a5090 legacy=0x7f749c1373b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.62:6789/0 2024-04-19T01:42:42.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:42.734+0000 7f749affd640 1 --2- 172.21.15.62:0/2596842009 >> v2:172.21.15.71:3300/0 conn(0x7f749c0724f0 0x7f749c0a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:42.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:42.734+0000 7f749affd640 10 monclient(hunting): get_auth_request con 0x7f749c0724f0 auth_method 0 2024-04-19T01:42:42.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:42.734+0000 7f749affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:42.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:42.734+0000 7f749affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:42.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:42.734+0000 7f749affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:42:42.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:42.734+0000 7f749affd640 1 -- 172.21.15.62:0/2596842009 >> v2:172.21.15.71:3300/0 conn(0x7f749c0724f0 msgr2=0x7f749c0a4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:42:42.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:42.734+0000 7f749affd640 1 -- 172.21.15.62:0/2596842009 >> v2:172.21.15.71:3300/0 conn(0x7f749c0724f0 msgr2=0x7f749c0a4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:42:42.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:42.734+0000 7f749affd640 1 --2- 172.21.15.62:0/2596842009 >> v2:172.21.15.71:3300/0 conn(0x7f749c0724f0 0x7f749c0a4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:42:42.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:42.734+0000 7f749affd640 1 --2- 172.21.15.62:0/2596842009 >> v2:172.21.15.71:3300/0 conn(0x7f749c0724f0 0x7f749c0a4b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:42:44.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.318+0000 7f7498ff9640 10 monclient(hunting): tick 2024-04-19T01:42:44.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.318+0000 7f7498ff9640 10 monclient(hunting): _check_auth_tickets 2024-04-19T01:42:44.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.318+0000 7f7498ff9640 1 monclient(hunting): continuing hunt 2024-04-19T01:42:44.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.318+0000 7f7498ff9640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T01:42:44.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.318+0000 7f7498ff9640 1 -- 172.21.15.62:0/2596842009 >> v2:172.21.15.71:3300/0 conn(0x7f749c0724f0 msgr2=0x7f749c0a4b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:42:44.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.318+0000 7f7498ff9640 1 --2- 172.21.15.62:0/2596842009 >> v2:172.21.15.71:3300/0 conn(0x7f749c0724f0 0x7f749c0a4b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:44.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.322+0000 7f7498ff9640 1 -- 172.21.15.62:0/2596842009 >> v1:172.21.15.71:6789/0 conn(0x7f749c137ac0 legacy=0x7f749c13f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:44.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.322+0000 7f7498ff9640 1 -- 172.21.15.62:0/2596842009 >> v1:172.21.15.62:6789/0 conn(0x7f749c0a5090 legacy=0x7f749c1373b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:42:44.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.322+0000 7f7498ff9640 10 monclient: _add_conns ranks=[3,2] 2024-04-19T01:42:44.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.322+0000 7f7498ff9640 1 --2- 172.21.15.62:0/2596842009 >> v2:172.21.15.62:3300/0 conn(0x7f7488003740 0x7f7488005ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:44.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.322+0000 7f7498ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f7488003740 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:44.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.322+0000 7f7498ff9640 1 --2- 172.21.15.62:0/2596842009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7488006230 0x7f74880086f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:44.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.322+0000 7f7498ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f7488006230 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:44.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.322+0000 7f7498ff9640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:44.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.322+0000 7f7498ff9640 1 -- 172.21.15.62:0/2596842009 --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f749c09c500 con 0x7f7488003740 2024-04-19T01:42:44.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.322+0000 7f7498ff9640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:44.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.322+0000 7f7498ff9640 1 -- 172.21.15.62:0/2596842009 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f7488008dd0 con 0x7f7488006230 2024-04-19T01:42:44.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.322+0000 7f7498ff9640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:44.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.322+0000 7f749affd640 1 -- 172.21.15.62:0/2596842009 >> v2:172.21.15.62:3300/0 conn(0x7f7488003740 msgr2=0x7f7488005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.62:3300/0 2024-04-19T01:42:44.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.322+0000 7f749affd640 1 --2- 172.21.15.62:0/2596842009 >> v2:172.21.15.62:3300/0 conn(0x7f7488003740 0x7f7488005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:42:44.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.322+0000 7f749a7fc640 1 --2- 172.21.15.62:0/2596842009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7488006230 0x7f74880086f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:44.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.322+0000 7f749a7fc640 10 monclient(hunting): get_auth_request con 0x7f7488006230 auth_method 0 2024-04-19T01:42:44.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.322+0000 7f749a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:44.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.322+0000 7f749a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:44.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.322+0000 7f749a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:44.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.322+0000 7f749a7fc640 1 -- 172.21.15.62:0/2596842009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7488006230 msgr2=0x7f74880086f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:42:44.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.322+0000 7f749a7fc640 1 -- 172.21.15.62:0/2596842009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7488006230 msgr2=0x7f74880086f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:42:44.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.322+0000 7f749a7fc640 1 --2- 172.21.15.62:0/2596842009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7488006230 0x7f74880086f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:42:44.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.322+0000 7f749a7fc640 1 --2- 172.21.15.62:0/2596842009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7488006230 0x7f74880086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:42:44.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.526+0000 7f749affd640 1 -- 172.21.15.62:0/2596842009 >> v2:172.21.15.62:3300/0 conn(0x7f7488003740 msgr2=0x7f7488005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.62:3300/0 2024-04-19T01:42:44.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.526+0000 7f749affd640 1 --2- 172.21.15.62:0/2596842009 >> v2:172.21.15.62:3300/0 conn(0x7f7488003740 0x7f7488005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:42:44.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.526+0000 7f749a7fc640 1 --2- 172.21.15.62:0/2596842009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7488006230 0x7f74880086f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:44.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.526+0000 7f749a7fc640 10 monclient(hunting): get_auth_request con 0x7f7488006230 auth_method 0 2024-04-19T01:42:44.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.526+0000 7f749a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:44.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.526+0000 7f749a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:44.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.526+0000 7f749a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:42:44.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.526+0000 7f749a7fc640 1 -- 172.21.15.62:0/2596842009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7488006230 msgr2=0x7f74880086f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:42:44.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.526+0000 7f749a7fc640 1 -- 172.21.15.62:0/2596842009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7488006230 msgr2=0x7f74880086f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:42:44.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.526+0000 7f749a7fc640 1 --2- 172.21.15.62:0/2596842009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7488006230 0x7f74880086f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:42:44.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.526+0000 7f749a7fc640 1 --2- 172.21.15.62:0/2596842009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7488006230 0x7f74880086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:42:44.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.930+0000 7f749affd640 1 -- 172.21.15.62:0/2596842009 >> v2:172.21.15.62:3300/0 conn(0x7f7488003740 msgr2=0x7f7488005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.62:3300/0 2024-04-19T01:42:44.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.930+0000 7f749affd640 1 --2- 172.21.15.62:0/2596842009 >> v2:172.21.15.62:3300/0 conn(0x7f7488003740 0x7f7488005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:42:44.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.930+0000 7f749a7fc640 1 --2- 172.21.15.62:0/2596842009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7488006230 0x7f74880086f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:44.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.930+0000 7f749a7fc640 10 monclient(hunting): get_auth_request con 0x7f7488006230 auth_method 0 2024-04-19T01:42:44.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.930+0000 7f749a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:44.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.930+0000 7f749a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:44.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.930+0000 7f749a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:42:44.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.930+0000 7f749a7fc640 1 -- 172.21.15.62:0/2596842009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7488006230 msgr2=0x7f74880086f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:42:44.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.930+0000 7f749a7fc640 1 -- 172.21.15.62:0/2596842009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7488006230 msgr2=0x7f74880086f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:42:44.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.930+0000 7f749a7fc640 1 --2- 172.21.15.62:0/2596842009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7488006230 0x7f74880086f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:42:44.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:44.930+0000 7f749a7fc640 1 --2- 172.21.15.62:0/2596842009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7488006230 0x7f74880086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:42:45.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.734+0000 7f749affd640 1 -- 172.21.15.62:0/2596842009 >> v2:172.21.15.62:3300/0 conn(0x7f7488003740 msgr2=0x7f7488005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.62:3300/0 2024-04-19T01:42:45.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.734+0000 7f749affd640 1 --2- 172.21.15.62:0/2596842009 >> v2:172.21.15.62:3300/0 conn(0x7f7488003740 0x7f7488005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:42:45.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.734+0000 7f749a7fc640 1 --2- 172.21.15.62:0/2596842009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7488006230 0x7f74880086f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:45.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.734+0000 7f749a7fc640 10 monclient(hunting): get_auth_request con 0x7f7488006230 auth_method 0 2024-04-19T01:42:45.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.734+0000 7f749a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:45.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.734+0000 7f749a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:45.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.734+0000 7f749a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:42:45.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.734+0000 7f749a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:45.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.734+0000 7f749a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:45.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.734+0000 7f749a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:45.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.734+0000 7f749a7fc640 10 monclient(hunting): handle_auth_done global_id 4382 payload 274 2024-04-19T01:42:45.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.734+0000 7f749a7fc640 1 -- 172.21.15.62:0/2596842009 >> v2:172.21.15.62:3300/0 conn(0x7f7488003740 msgr2=0x7f7488005ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:42:45.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.734+0000 7f749a7fc640 1 --2- 172.21.15.62:0/2596842009 >> v2:172.21.15.62:3300/0 conn(0x7f7488003740 0x7f7488005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:45.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.734+0000 7f749a7fc640 10 monclient: _finish_hunting 0 2024-04-19T01:42:45.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.734+0000 7f749a7fc640 1 monclient: found mon.noname-b 2024-04-19T01:42:45.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.734+0000 7f749a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:45.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.734+0000 7f749a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:42:45.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.734+0000 7f749a7fc640 1 -- 172.21.15.62:0/2596842009 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f749c09bc60 con 0x7f7488006230 2024-04-19T01:42:45.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.738+0000 7f749a7fc640 1 -- 172.21.15.62:0/2596842009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7488006230 msgr2=0x7f74880086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:45.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.738+0000 7f749a7fc640 1 --2- 172.21.15.62:0/2596842009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7488006230 0x7f74880086f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7490003990 tx=0x7f749001d1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=678e791357256af5 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:45.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.738+0000 7f7499ffb640 1 -- 172.21.15.62:0/2596842009 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74900b1070 con 0x7f7488006230 2024-04-19T01:42:45.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.738+0000 7f7499ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:45.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.738+0000 7f7499ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:42:45.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.738+0000 7f7499ffb640 10 monclient: dump: 2024-04-19T01:42:45.741 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:45.741 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:45.741 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:45.741 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:45.741 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:45.741 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:45.741 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:45.741 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:45.741 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:45.741 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:45.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.738+0000 7f7499ffb640 10 monclient: _finish_auth 0 2024-04-19T01:42:45.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.738+0000 7f7499ffb640 10 monclient: _check_auth_tickets 2024-04-19T01:42:45.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.738+0000 7f7499ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:45.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.738+0000 7f7499ffb640 1 -- 172.21.15.62:0/2596842009 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74900ac600 con 0x7f7488006230 2024-04-19T01:42:45.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.738+0000 7f7499ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:45.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.738+0000 7f7499ffb640 1 -- 172.21.15.62:0/2596842009 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74900ac9d0 con 0x7f7488006230 2024-04-19T01:42:45.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.738+0000 7f7499ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:45.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.738+0000 7f7499ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:45.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.738+0000 7f7499ffb640 10 monclient: dump: 2024-04-19T01:42:45.742 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:45.742 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:45.742 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:45.742 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:45.742 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:45.742 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:45.742 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:45.743 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:45.743 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:45.743 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:45.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.738+0000 7f74a1539640 5 monclient: authenticate success, global_id 4382 2024-04-19T01:42:45.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.738+0000 7f74a1539640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:45.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.738+0000 7f74a1539640 4 set_mon_vals no callback set 2024-04-19T01:42:45.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.742+0000 7f74a1539640 10 monclient: shutdown 2024-04-19T01:42:45.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.742+0000 7f74a1539640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:45.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.742+0000 7f74a1539640 1 -- 172.21.15.62:0/2596842009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7488006230 msgr2=0x7f74880086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:45.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.742+0000 7f74a1539640 1 --2- 172.21.15.62:0/2596842009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7488006230 0x7f74880086f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7490003990 tx=0x7f749001d1a0 comp rx=0 tx=0).stop 2024-04-19T01:42:45.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.742+0000 7f749b7fe640 1 -- 172.21.15.62:0/2596842009 reap_dead start 2024-04-19T01:42:45.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.742+0000 7f74a1539640 1 -- 172.21.15.62:0/2596842009 shutdown_connections 2024-04-19T01:42:45.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.742+0000 7f74a1539640 1 -- 172.21.15.62:0/2596842009 >> 172.21.15.62:0/2596842009 conn(0x7f749c0a9e80 msgr2=0x7f749c0aa290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:45.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.742+0000 7f74a1539640 1 -- 172.21.15.62:0/2596842009 shutdown_connections 2024-04-19T01:42:45.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.742+0000 7f74a1539640 1 -- 172.21.15.62:0/2596842009 wait complete. 2024-04-19T01:42:45.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.742+0000 7f74a1539640 10 monclient: build_initial_monmap 2024-04-19T01:42:45.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.742+0000 7f74a1539640 10 monclient: monmap: 2024-04-19T01:42:45.745 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:45.745 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:45.745 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:45.745 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:45.745 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:45.745 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:45.745 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:45.745 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:45.745 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:45.745 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:45.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.742+0000 7f74a1539640 1 Processor -- start 2024-04-19T01:42:45.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.742+0000 7f74a1539640 1 -- start start 2024-04-19T01:42:45.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.742+0000 7f74a1539640 10 monclient: init 2024-04-19T01:42:45.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.742+0000 7f74a1539640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:45.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.742+0000 7f74a1539640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-19T01:42:45.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74a1539640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f749c138340 0x7f749c138720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:45.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74a1539640 10 monclient(hunting): picked mon.noname-c con 0x7f749c138340 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:45.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74a1539640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f749c13b790 0x7f749c138c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:45.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74a1539640 10 monclient(hunting): picked mon.noname-b con 0x7f749c13b790 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:45.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74a1539640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f749c1311e0 0x7f749c131670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:45.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74a1539640 10 monclient(hunting): picked mon.noname-a con 0x7f749c1311e0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:45.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74a1539640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:45.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74a1539640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f749c0a54a0 con 0x7f749c1311e0 2024-04-19T01:42:45.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74a1539640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:45.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74a1539640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f749c0a6260 con 0x7f749c13b790 2024-04-19T01:42:45.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74a1539640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:45.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74a1539640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f749c143630 con 0x7f749c138340 2024-04-19T01:42:45.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74a1539640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:45.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74a1539640 10 monclient(hunting): authenticate will time out at 951.568272s 2024-04-19T01:42:45.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f749b7fe640 1 -- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f749c1311e0 msgr2=0x7f749c131670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:3300/0 2024-04-19T01:42:45.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f749affd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f749c138340 0x7f749c138720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:45.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f749b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f749c1311e0 0x7f749c131670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:42:45.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f749affd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f749c138340 0x7f749c138720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:59162/0 (socket says 172.21.15.62:59162) 2024-04-19T01:42:45.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f749affd640 1 -- 172.21.15.62:0/738711092 learned_addr learned my addr 172.21.15.62:0/738711092 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:45.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f749affd640 10 monclient(hunting): get_auth_request con 0x7f749c138340 auth_method 0 2024-04-19T01:42:45.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f749affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:45.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f749affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:45.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f749affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:45.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f749affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:45.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f749affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:45.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f749affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:45.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f749affd640 10 monclient(hunting): handle_auth_done global_id 4385 payload 824 2024-04-19T01:42:45.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f749a7fc640 1 --2- 172.21.15.62:0/738711092 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f749c13b790 0x7f749c138c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:45.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f749affd640 1 -- 172.21.15.62:0/738711092 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f749c13b790 msgr2=0x7f749c138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:45.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f749affd640 1 --2- 172.21.15.62:0/738711092 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f749c13b790 0x7f749c138c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:45.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f749affd640 1 -- 172.21.15.62:0/738711092 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f749c1311e0 msgr2=0x7f749c131670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:42:45.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f749affd640 1 --2- 172.21.15.62:0/738711092 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f749c1311e0 0x7f749c131670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:45.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f749affd640 10 monclient: _finish_hunting 0 2024-04-19T01:42:45.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f749affd640 1 monclient: found mon.noname-c 2024-04-19T01:42:45.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f749affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:45.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f749affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:42:45.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f749affd640 1 -- 172.21.15.62:0/738711092 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f749c1391a0 con 0x7f749c138340 2024-04-19T01:42:45.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f749a7fc640 10 monclient: get_auth_request con 0x7f749c13b790 auth_method 0 2024-04-19T01:42:45.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f749a7fc640 1 --2- 172.21.15.62:0/738711092 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f749c13b790 0x7f749c138c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:42:45.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f749affd640 1 -- 172.21.15.62:0/738711092 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f749c138340 msgr2=0x7f749c138720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:45.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f749affd640 1 --2- 172.21.15.62:0/738711092 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f749c138340 0x7f749c138720 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f748c014520 tx=0x7f748c0149e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:45.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74837fe640 1 -- 172.21.15.62:0/738711092 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f748c01b030 con 0x7f749c138340 2024-04-19T01:42:45.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:45.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74837fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:45.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74837fe640 10 monclient: dump: 2024-04-19T01:42:45.753 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:45.753 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:45.753 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:45.754 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:45.754 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:45.754 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:45.754 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:45.754 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:45.754 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:45.754 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:45.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74837fe640 10 monclient: _finish_auth 0 2024-04-19T01:42:45.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74837fe640 10 monclient: _check_auth_tickets 2024-04-19T01:42:45.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:45.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74837fe640 1 -- 172.21.15.62:0/738711092 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f748c010a50 con 0x7f749c138340 2024-04-19T01:42:45.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74837fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:45.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74837fe640 1 -- 172.21.15.62:0/738711092 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f748c00f040 con 0x7f749c138340 2024-04-19T01:42:45.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:45.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:45.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74837fe640 10 monclient: dump: 2024-04-19T01:42:45.755 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:45.755 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:45.755 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:45.755 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:45.755 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:45.755 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:45.755 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:45.755 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:45.755 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:45.756 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:45.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74997fa640 4 set_mon_vals no callback set 2024-04-19T01:42:45.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74a1539640 5 monclient: authenticate success, global_id 4385 2024-04-19T01:42:45.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74a1539640 10 monclient: _renew_subs 2024-04-19T01:42:45.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74a1539640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:45.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74a1539640 1 -- 172.21.15.62:0/738711092 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f749c1407f0 con 0x7f749c138340 2024-04-19T01:42:45.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74a1539640 10 monclient: _renew_subs 2024-04-19T01:42:45.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74a1539640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:45.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.746+0000 7f74a1539640 1 -- 172.21.15.62:0/738711092 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f749c0a1290 con 0x7f749c138340 2024-04-19T01:42:45.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.750+0000 7f74837fe640 1 -- 172.21.15.62:0/738711092 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f748c01e030 con 0x7f749c138340 2024-04-19T01:42:45.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.750+0000 7f74a1539640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:45.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.750+0000 7f74a1539640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:45.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.750+0000 7f74a1539640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:45.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.750+0000 7f74a1539640 1 -- 172.21.15.62:0/738711092 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f749c0692b0 con 0x7f749c138340 2024-04-19T01:42:45.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.750+0000 7f74837fe640 1 --2- 172.21.15.62:0/738711092 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f7488037320 0x7f74880397e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:45.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.750+0000 7f74837fe640 1 -- 172.21.15.62:0/738711092 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 4738+0+0 (secure 0 0 0) 0x7f748c0499b0 con 0x7f749c138340 2024-04-19T01:42:45.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.750+0000 7f749a7fc640 1 --2- 172.21.15.62:0/738711092 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f7488037320 0x7f74880397e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:45.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.750+0000 7f749a7fc640 10 monclient: get_auth_request con 0x7f7488037320 auth_method 0 2024-04-19T01:42:45.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.754+0000 7f749a7fc640 1 --2- 172.21.15.62:0/738711092 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f7488037320 0x7f74880397e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f74900ac000 tx=0x7f74900a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:45.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.754+0000 7f74837fe640 1 -- 172.21.15.62:0/738711092 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f748c028350 con 0x7f749c138340 2024-04-19T01:42:45.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.754+0000 7f74837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:45.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.754+0000 7f74837fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:45.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.982+0000 7f74a1539640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:42:45.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.982+0000 7f74a1539640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:42:45.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.982+0000 7f74a1539640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:45.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.982+0000 7f74a1539640 1 -- 172.21.15.62:0/738711092 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f749c136d50 con 0x7f749c138340 2024-04-19T01:42:45.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.986+0000 7f74837fe640 1 -- 172.21.15.62:0/738711092 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f748c0483d0 con 0x7f749c138340 2024-04-19T01:42:45.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.986+0000 7f74837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:42:45.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.986+0000 7f74837fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:45.986 INFO:teuthology.orchestra.run.smithi062.stdout:{"election_epoch":10,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","modified":"2024-04-19T01:41:38.497795Z","created":"2024-04-19T01:41:38.497795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:42:45.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.986+0000 7f74a1539640 1 -- 172.21.15.62:0/738711092 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f7488037320 msgr2=0x7f74880397e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:45.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.986+0000 7f74a1539640 1 --2- 172.21.15.62:0/738711092 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f7488037320 0x7f74880397e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f74900ac000 tx=0x7f74900a7000 comp rx=0 tx=0).stop 2024-04-19T01:42:45.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.986+0000 7f74a1539640 10 monclient: shutdown 2024-04-19T01:42:45.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.986+0000 7f74a1539640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:45.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.986+0000 7f74a1539640 1 -- 172.21.15.62:0/738711092 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f749c138340 msgr2=0x7f749c138720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:45.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.986+0000 7f74a1539640 1 --2- 172.21.15.62:0/738711092 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f749c138340 0x7f749c138720 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f748c014520 tx=0x7f748c0149e0 comp rx=0 tx=0).stop 2024-04-19T01:42:45.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.986+0000 7f74a1539640 1 -- 172.21.15.62:0/738711092 shutdown_connections 2024-04-19T01:42:45.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.986+0000 7f74a1539640 1 --2- 172.21.15.62:0/738711092 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f749c1311e0 0x7f749c131670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:45.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.986+0000 7f74a1539640 1 --2- 172.21.15.62:0/738711092 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f749c13b790 0x7f749c138c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:45.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.986+0000 7f74a1539640 1 --2- 172.21.15.62:0/738711092 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f7488037320 0x7f74880397e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:45.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.986+0000 7f74a1539640 1 --2- 172.21.15.62:0/738711092 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f749c138340 0x7f749c138720 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:45.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.986+0000 7f74a1539640 1 -- 172.21.15.62:0/738711092 >> 172.21.15.62:0/738711092 conn(0x7f749c0a9e80 msgr2=0x7f749c0a8fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:45.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.986+0000 7f74a1539640 1 -- 172.21.15.62:0/738711092 shutdown_connections 2024-04-19T01:42:45.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:45.986+0000 7f74a1539640 1 -- 172.21.15.62:0/738711092 wait complete. 2024-04-19T01:42:46.005 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-19T01:42:46.005 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-19T01:42:46.005 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-19T01:42:46.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2dba22f640 10 monclient: get_monmap_and_config 2024-04-19T01:42:46.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2dba22f640 10 monclient: build_initial_monmap 2024-04-19T01:42:46.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2dba22f640 10 monclient: monmap: 2024-04-19T01:42:46.123 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:46.123 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:46.123 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:46.127209+0000 2024-04-19T01:42:46.123 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:46.127209+0000 2024-04-19T01:42:46.123 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:46.123 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:46.123 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:46.123 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:46.123 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:46.123 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:46.123 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:46.123 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:46.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2dba22f640 1 Processor -- start 2024-04-19T01:42:46.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2dba22f640 1 -- start start 2024-04-19T01:42:46.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2dba22f640 10 monclient: init 2024-04-19T01:42:46.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2dba22f640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:46.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2dba22f640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-04-19T01:42:46.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2dba22f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2db40a47e0 addr v1:172.21.15.71:6789/0 2024-04-19T01:42:46.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2dba22f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2db40a4bc0 0x7f2db40a4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:46.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2dba22f640 10 monclient(hunting): picked mon.noname-a con 0x7f2db40a4bc0 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:46.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2dba22f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2db4139560 0x7f2db413b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:46.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2dba22f640 10 monclient(hunting): picked mon.noname-b con 0x7f2db4139560 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:46.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2dba22f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2db409e6e0 con 0x7f2db40a47e0 2024-04-19T01:42:46.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2dba22f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:46.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2dba22f640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f2db409d4e0 con 0x7f2db40a4bc0 2024-04-19T01:42:46.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2dba22f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:46.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2db2ffd640 1 -- >> v2:172.21.15.62:3300/0 conn(0x7f2db40a4bc0 msgr2=0x7f2db40a4ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.62:3300/0 2024-04-19T01:42:46.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2dba22f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f2db409e9c0 con 0x7f2db4139560 2024-04-19T01:42:46.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2dba22f640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:46.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2dba22f640 10 monclient(hunting): authenticate will time out at 951.947260s 2024-04-19T01:42:46.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2db2ffd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2db40a4bc0 0x7f2db40a4ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:42:46.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2db3fff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2db4139560 0x7f2db413b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:46.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2db3fff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2db4139560 0x7f2db413b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:59176/0 (socket says 172.21.15.62:59176) 2024-04-19T01:42:46.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2db3fff640 1 -- 172.21.15.62:0/2690997370 learned_addr learned my addr 172.21.15.62:0/2690997370 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:46.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2db3fff640 10 monclient(hunting): get_auth_request con 0x7f2db4139560 auth_method 0 2024-04-19T01:42:46.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2db3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:46.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2db3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:46.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2db3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:46.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2db3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:46.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2db3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:46.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2db3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:46.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2db3fff640 10 monclient(hunting): handle_auth_done global_id 4388 payload 274 2024-04-19T01:42:46.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2db3fff640 1 -- 172.21.15.62:0/2690997370 >> v2:172.21.15.62:3300/0 conn(0x7f2db40a4bc0 msgr2=0x7f2db40a4ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:42:46.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2db3fff640 1 --2- 172.21.15.62:0/2690997370 >> v2:172.21.15.62:3300/0 conn(0x7f2db40a4bc0 0x7f2db40a4ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:46.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2db3fff640 1 -- 172.21.15.62:0/2690997370 >> v1:172.21.15.71:6789/0 conn(0x7f2db40a47e0 legacy=0x7f2db409f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:46.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2db3fff640 10 monclient: _finish_hunting 0 2024-04-19T01:42:46.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2db3fff640 1 monclient: found mon.noname-b 2024-04-19T01:42:46.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2db3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:46.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2db3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:42:46.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.122+0000 7f2db3fff640 1 -- 172.21.15.62:0/2690997370 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2db40a5530 con 0x7f2db4139560 2024-04-19T01:42:46.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.126+0000 7f2db3fff640 1 -- 172.21.15.62:0/2690997370 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2db4139560 msgr2=0x7f2db413b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:46.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.126+0000 7f2db3fff640 1 --2- 172.21.15.62:0/2690997370 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2db4139560 0x7f2db413b950 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2dac009e50 tx=0x7f2dac01c9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=244046addebfdf22 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:46.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db27fc640 1 -- 172.21.15.62:0/2690997370 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dac0a55e0 con 0x7f2db4139560 2024-04-19T01:42:46.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:46.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:42:46.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db27fc640 10 monclient: dump: 2024-04-19T01:42:46.130 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:46.130 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:46.130 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:46.130 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:46.130 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:46.130 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:46.130 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:46.130 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:46.130 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:46.130 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:46.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db27fc640 10 monclient: _finish_auth 0 2024-04-19T01:42:46.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db27fc640 10 monclient: _check_auth_tickets 2024-04-19T01:42:46.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:46.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db27fc640 1 -- 172.21.15.62:0/2690997370 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dac0a5780 con 0x7f2db4139560 2024-04-19T01:42:46.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 5 monclient: authenticate success, global_id 4388 2024-04-19T01:42:46.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:42:46.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db27fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:46.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:46.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 4 set_mon_vals no callback set 2024-04-19T01:42:46.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db27fc640 1 -- 172.21.15.62:0/2690997370 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dac0a5b50 con 0x7f2db4139560 2024-04-19T01:42:46.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:46.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:46.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db27fc640 10 monclient: dump: 2024-04-19T01:42:46.131 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:46.132 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:46.132 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:46.132 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:46.132 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:46.132 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:46.132 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:46.132 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:46.132 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:46.132 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:46.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 10 monclient: shutdown 2024-04-19T01:42:46.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:46.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 1 -- 172.21.15.62:0/2690997370 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2db4139560 msgr2=0x7f2db413b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:46.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 1 --2- 172.21.15.62:0/2690997370 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2db4139560 0x7f2db413b950 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2dac009e50 tx=0x7f2dac01c9a0 comp rx=0 tx=0).stop 2024-04-19T01:42:46.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 1 -- 172.21.15.62:0/2690997370 shutdown_connections 2024-04-19T01:42:46.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 1 --2- 172.21.15.62:0/2690997370 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2db4139560 0x7f2db413b950 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:46.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 1 --2- 172.21.15.62:0/2690997370 >> v2:172.21.15.62:3300/0 conn(0x7f2db40a4bc0 0x7f2db40a4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:46.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 1 -- 172.21.15.62:0/2690997370 >> 172.21.15.62:0/2690997370 conn(0x7f2db40a23d0 msgr2=0x7f2db40a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:46.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 1 -- 172.21.15.62:0/2690997370 shutdown_connections 2024-04-19T01:42:46.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 1 -- 172.21.15.62:0/2690997370 wait complete. 2024-04-19T01:42:46.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 10 monclient: build_initial_monmap 2024-04-19T01:42:46.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 10 monclient: monmap: 2024-04-19T01:42:46.140 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:46.140 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:46.140 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:46.140 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:46.140 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:46.140 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:46.141 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:46.141 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:46.141 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:46.141 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:46.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 1 Processor -- start 2024-04-19T01:42:46.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 1 -- start start 2024-04-19T01:42:46.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 10 monclient: init 2024-04-19T01:42:46.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:46.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-19T01:42:46.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2db40a47e0 0x7f2db41cf0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:46.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 10 monclient(hunting): picked mon.noname-b con 0x7f2db40a47e0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:46.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2db40a4bc0 0x7f2db4134fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:46.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 10 monclient(hunting): picked mon.noname-c con 0x7f2db40a4bc0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:46.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2db4139560 0x7f2db41354f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:46.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 10 monclient(hunting): picked mon.noname-a con 0x7f2db4139560 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:46.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:46.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2db413ea40 con 0x7f2db4139560 2024-04-19T01:42:46.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:46.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f2db413e8c0 con 0x7f2db40a47e0 2024-04-19T01:42:46.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:46.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f2db413ebc0 con 0x7f2db40a4bc0 2024-04-19T01:42:46.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:46.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2dba22f640 10 monclient(hunting): authenticate will time out at 951.954327s 2024-04-19T01:42:46.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db3fff640 1 -- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2db4139560 msgr2=0x7f2db41354f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:3300/0 2024-04-19T01:42:46.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2db4139560 0x7f2db41354f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:42:46.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db2ffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2db40a4bc0 0x7f2db4134fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:46.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db2ffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2db40a4bc0 0x7f2db4134fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:59182/0 (socket says 172.21.15.62:59182) 2024-04-19T01:42:46.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db2ffd640 1 -- 172.21.15.62:0/2735091876 learned_addr learned my addr 172.21.15.62:0/2735091876 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:46.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db2ffd640 10 monclient(hunting): get_auth_request con 0x7f2db40a4bc0 auth_method 0 2024-04-19T01:42:46.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:46.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:46.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:46.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:46.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:46.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:46.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db37fe640 1 --2- 172.21.15.62:0/2735091876 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2db40a47e0 0x7f2db41cf0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:46.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db2ffd640 10 monclient(hunting): handle_auth_done global_id 4391 payload 824 2024-04-19T01:42:46.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db2ffd640 1 -- 172.21.15.62:0/2735091876 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2db40a47e0 msgr2=0x7f2db41cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:46.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db2ffd640 1 --2- 172.21.15.62:0/2735091876 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2db40a47e0 0x7f2db41cf0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:46.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db2ffd640 1 -- 172.21.15.62:0/2735091876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2db4139560 msgr2=0x7f2db41354f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:42:46.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db2ffd640 1 --2- 172.21.15.62:0/2735091876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2db4139560 0x7f2db41354f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:46.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db2ffd640 10 monclient: _finish_hunting 0 2024-04-19T01:42:46.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db2ffd640 1 monclient: found mon.noname-c 2024-04-19T01:42:46.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:46.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:42:46.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db2ffd640 1 -- 172.21.15.62:0/2735091876 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2db4121400 con 0x7f2db40a4bc0 2024-04-19T01:42:46.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db37fe640 10 monclient: get_auth_request con 0x7f2db40a47e0 auth_method 0 2024-04-19T01:42:46.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.130+0000 7f2db37fe640 1 --2- 172.21.15.62:0/2735091876 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2db40a47e0 0x7f2db41cf0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:42:46.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2db2ffd640 1 -- 172.21.15.62:0/2735091876 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2db40a4bc0 msgr2=0x7f2db4134fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:46.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2db2ffd640 1 --2- 172.21.15.62:0/2735091876 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2db40a4bc0 0x7f2db4134fb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2da800b580 tx=0x7f2da800ba40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:46.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2db2ffd640 1 --2- 172.21.15.62:0/2735091876 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2db40a4bc0 0x7f2db4134fb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2da800b580 tx=0x7f2da800ba40 comp rx=0 tx=0).queue_received will delay after 1.71349e+09 on 0x7f2da80050d0 config(0 keys) 2024-04-19T01:42:46.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2d9bfff640 1 -- 172.21.15.62:0/2735091876 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da8016070 con 0x7f2db40a4bc0 2024-04-19T01:42:46.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2d9bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:46.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2d9bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:46.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2d9bfff640 10 monclient: dump: 2024-04-19T01:42:46.145 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:46.145 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:46.145 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:46.145 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:46.145 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:46.145 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:46.145 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:46.145 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:46.146 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:46.146 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:46.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2d9bfff640 10 monclient: _finish_auth 0 2024-04-19T01:42:46.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2d9bfff640 10 monclient: _check_auth_tickets 2024-04-19T01:42:46.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2d9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:46.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2d9bfff640 1 -- 172.21.15.62:0/2735091876 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2da80050d0 con 0x7f2db40a4bc0 2024-04-19T01:42:46.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2d9bfff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:46.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2dba22f640 5 monclient: authenticate success, global_id 4391 2024-04-19T01:42:46.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2dba22f640 10 monclient: _renew_subs 2024-04-19T01:42:46.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2dba22f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:46.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2dba22f640 1 -- 172.21.15.62:0/2735091876 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2db409e050 con 0x7f2db40a4bc0 2024-04-19T01:42:46.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2db1ffb640 4 set_mon_vals no callback set 2024-04-19T01:42:46.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2dba22f640 10 monclient: _renew_subs 2024-04-19T01:42:46.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2dba22f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:46.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2dba22f640 1 -- 172.21.15.62:0/2735091876 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2db4122970 con 0x7f2db40a4bc0 2024-04-19T01:42:46.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2dba22f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:46.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2dba22f640 1 --2- 172.21.15.62:0/2735091876 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2db411e9e0 0x7f2db411edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:46.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2dba22f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:46.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2dba22f640 1 -- 172.21.15.62:0/2735091876 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f2db40a0660 con 0x7f2db411e9e0 2024-04-19T01:42:46.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2db37fe640 1 --2- 172.21.15.62:0/2735091876 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2db411e9e0 0x7f2db411edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:46.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2db37fe640 10 monclient: get_auth_request con 0x7f2db411e9e0 auth_method 0 2024-04-19T01:42:46.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2db37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:46.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2db37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:46.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2db37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:46.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2d9bfff640 1 -- 172.21.15.62:0/2735091876 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da80054c0 con 0x7f2db40a4bc0 2024-04-19T01:42:46.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2d9bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:46.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2d9bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:46.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2d9bfff640 10 monclient: dump: 2024-04-19T01:42:46.147 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:46.148 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:46.148 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:46.148 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:46.148 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:46.148 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:46.148 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:46.148 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:46.148 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:46.148 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:46.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2d9bfff640 1 -- 172.21.15.62:0/2735091876 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f2da8019070 con 0x7f2db40a4bc0 2024-04-19T01:42:46.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2db37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:46.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2db37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:46.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2db37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:46.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2d9bfff640 1 --2- 172.21.15.62:0/2735091876 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f2da00363d0 0x7f2da0038890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:46.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2db37fe640 10 monclient(hunting): handle_auth_done global_id 4394 payload 824 2024-04-19T01:42:46.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2db37fe640 1 -- 172.21.15.62:0/2735091876 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2db40692b0 con 0x7f2db411e9e0 2024-04-19T01:42:46.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2db3fff640 1 --2- 172.21.15.62:0/2735091876 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f2da00363d0 0x7f2da0038890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:46.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.134+0000 7f2db3fff640 10 monclient: get_auth_request con 0x7f2da00363d0 auth_method 0 2024-04-19T01:42:46.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.138+0000 7f2db37fe640 1 -- 172.21.15.62:0/2735091876 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2db411e9e0 msgr2=0x7f2db411edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:46.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.138+0000 7f2db37fe640 1 --2- 172.21.15.62:0/2735091876 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2db411e9e0 0x7f2db411edc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2da400d480 tx=0x7f2da400d940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:46.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.138+0000 7f2d9bfff640 1 -- 172.21.15.62:0/2735091876 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da4012a10 con 0x7f2db411e9e0 2024-04-19T01:42:46.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.138+0000 7f2d9bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:46.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.138+0000 7f2d9bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:46.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.138+0000 7f2d9bfff640 10 monclient: dump: 2024-04-19T01:42:46.149 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:46.149 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:46.149 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:46.149 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:46.149 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:46.149 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:46.149 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:46.149 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:46.149 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:46.150 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:46.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.138+0000 7f2d9bfff640 1 -- 172.21.15.62:0/2735091876 <== mon.2 v2:172.21.15.62:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2da4012a10 con 0x7f2db411e9e0 2024-04-19T01:42:46.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.138+0000 7f2d9bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:46.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.138+0000 7f2d9bfff640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:46.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.138+0000 7f2d9bfff640 1 -- 172.21.15.62:0/2735091876 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2db411e9e0 msgr2=0x7f2db411edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:46.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.138+0000 7f2d9bfff640 1 --2- 172.21.15.62:0/2735091876 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2db411e9e0 0x7f2db411edc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2da400d480 tx=0x7f2da400d940 comp rx=0 tx=0).stop 2024-04-19T01:42:46.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.138+0000 7f2d9bfff640 1 -- 172.21.15.62:0/2735091876 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2db411e9e0 msgr2=0x7f2db411edc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:42:46.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.138+0000 7f2d9bfff640 1 --2- 172.21.15.62:0/2735091876 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2db411e9e0 0x7f2db411edc0 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2da400d480 tx=0x7f2da400d940 comp rx=0 tx=0).stop 2024-04-19T01:42:46.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.138+0000 7f2db3fff640 1 --2- 172.21.15.62:0/2735091876 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f2da00363d0 0x7f2da0038890 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2db40762f0 tx=0x7f2dac0a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:46.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.142+0000 7f2dba22f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:42:46.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.142+0000 7f2dba22f640 1 --2- 172.21.15.62:0/2735091876 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2db411f300 0x7f2db4139940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:46.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.142+0000 7f2dba22f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:46.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.142+0000 7f2dba22f640 1 -- 172.21.15.62:0/2735091876 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f2db4122190 con 0x7f2db411f300 2024-04-19T01:42:46.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.142+0000 7f2db37fe640 1 --2- 172.21.15.62:0/2735091876 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2db411f300 0x7f2db4139940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:46.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.142+0000 7f2db37fe640 10 monclient: get_auth_request con 0x7f2db411f300 auth_method 0 2024-04-19T01:42:46.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.142+0000 7f2db37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:46.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.142+0000 7f2db37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:46.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.142+0000 7f2db37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:46.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.142+0000 7f2db37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:46.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.142+0000 7f2db37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:46.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.142+0000 7f2db37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:46.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.142+0000 7f2db37fe640 10 monclient(hunting): handle_auth_done global_id 4397 payload 824 2024-04-19T01:42:46.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.142+0000 7f2db37fe640 1 -- 172.21.15.62:0/2735091876 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2db41224b0 con 0x7f2db411f300 2024-04-19T01:42:46.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.146+0000 7f2db37fe640 1 -- 172.21.15.62:0/2735091876 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2db411f300 msgr2=0x7f2db4139940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:46.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.146+0000 7f2db37fe640 1 --2- 172.21.15.62:0/2735091876 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2db411f300 0x7f2db4139940 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2da400dee0 tx=0x7f2da4003770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:46.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.146+0000 7f2d9bfff640 1 -- 172.21.15.62:0/2735091876 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da4007450 con 0x7f2db411f300 2024-04-19T01:42:46.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.146+0000 7f2d9bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:46.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.146+0000 7f2d9bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:46.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.146+0000 7f2d9bfff640 10 monclient: dump: 2024-04-19T01:42:46.153 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:46.153 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:46.153 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:46.153 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:46.153 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:46.153 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:46.153 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:46.153 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:46.153 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:46.153 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:46.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.146+0000 7f2d9bfff640 1 -- 172.21.15.62:0/2735091876 <== mon.2 v2:172.21.15.62:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4705 (secure 0 0 0) 0x7f2da4007450 con 0x7f2db411f300 2024-04-19T01:42:46.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.146+0000 7f2d9bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:42:46.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.146+0000 7f2d9bfff640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:46.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.146+0000 7f2d9bfff640 1 -- 172.21.15.62:0/2735091876 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2db411f300 msgr2=0x7f2db4139940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:46.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.146+0000 7f2d9bfff640 1 --2- 172.21.15.62:0/2735091876 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2db411f300 0x7f2db4139940 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2da400dee0 tx=0x7f2da4003770 comp rx=0 tx=0).stop 2024-04-19T01:42:46.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.146+0000 7f2d9bfff640 1 -- 172.21.15.62:0/2735091876 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2db411f300 msgr2=0x7f2db4139940 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:42:46.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.146+0000 7f2d9bfff640 1 --2- 172.21.15.62:0/2735091876 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2db411f300 0x7f2db4139940 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2da400dee0 tx=0x7f2da4003770 comp rx=0 tx=0).stop 2024-04-19T01:42:46.154 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-19T01:42:46.154 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "c", 2024-04-19T01:42:46.155 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 2, 2024-04-19T01:42:46.155 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "peon", 2024-04-19T01:42:46.155 INFO:teuthology.orchestra.run.smithi062.stdout: "election_epoch": 10, 2024-04-19T01:42:46.155 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 42351, 2024-04-19T01:42:46.155 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum": [ 2024-04-19T01:42:46.155 INFO:teuthology.orchestra.run.smithi062.stdout: 1, 2024-04-19T01:42:46.155 INFO:teuthology.orchestra.run.smithi062.stdout: 2 2024-04-19T01:42:46.155 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:42:46.155 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_age": 1, 2024-04-19T01:42:46.155 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:42:46.155 INFO:teuthology.orchestra.run.smithi062.stdout: "required_con": "2450521708007866372", 2024-04-19T01:42:46.155 INFO:teuthology.orchestra.run.smithi062.stdout: "required_mon": [ 2024-04-19T01:42:46.155 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:42:46.155 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:42:46.155 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:42:46.156 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:42:46.156 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:42:46.156 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:42:46.156 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:42:46.156 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:42:46.156 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:42:46.156 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:42:46.156 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:42:46.156 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:42:46.156 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:42:46.156 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_mon": [ 2024-04-19T01:42:46.156 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:42:46.156 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:42:46.156 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:42:46.156 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:42:46.156 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:42:46.157 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:42:46.157 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:42:46.157 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:42:46.157 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:42:46.157 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:42:46.157 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:42:46.157 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:42:46.157 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:46.157 INFO:teuthology.orchestra.run.smithi062.stdout: "outside_quorum": [], 2024-04-19T01:42:46.157 INFO:teuthology.orchestra.run.smithi062.stdout: "extra_probe_peers": [], 2024-04-19T01:42:46.157 INFO:teuthology.orchestra.run.smithi062.stdout: "sync_provider": [], 2024-04-19T01:42:46.157 INFO:teuthology.orchestra.run.smithi062.stdout: "monmap": { 2024-04-19T01:42:46.157 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 1, 2024-04-19T01:42:46.157 INFO:teuthology.orchestra.run.smithi062.stdout: "fsid": "6466e4b6-2520-485f-95b3-9401915fbbad", 2024-04-19T01:42:46.157 INFO:teuthology.orchestra.run.smithi062.stdout: "modified": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:42:46.157 INFO:teuthology.orchestra.run.smithi062.stdout: "created": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:42:46.157 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release": 19, 2024-04-19T01:42:46.158 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release_name": "squid", 2024-04-19T01:42:46.158 INFO:teuthology.orchestra.run.smithi062.stdout: "election_strategy": 1, 2024-04-19T01:42:46.158 INFO:teuthology.orchestra.run.smithi062.stdout: "disallowed_leaders: ": "", 2024-04-19T01:42:46.158 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false, 2024-04-19T01:42:46.158 INFO:teuthology.orchestra.run.smithi062.stdout: "tiebreaker_mon": "", 2024-04-19T01:42:46.158 INFO:teuthology.orchestra.run.smithi062.stdout: "removed_ranks: ": "", 2024-04-19T01:42:46.158 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:42:46.158 INFO:teuthology.orchestra.run.smithi062.stdout: "persistent": [ 2024-04-19T01:42:46.158 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:42:46.158 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:42:46.158 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:42:46.158 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:42:46.158 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:42:46.158 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:42:46.158 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:42:46.159 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:42:46.159 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:42:46.159 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:42:46.159 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:42:46.159 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:42:46.159 INFO:teuthology.orchestra.run.smithi062.stdout: "optional": [] 2024-04-19T01:42:46.159 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:46.159 INFO:teuthology.orchestra.run.smithi062.stdout: "mons": [ 2024-04-19T01:42:46.159 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:46.159 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 0, 2024-04-19T01:42:46.159 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "a", 2024-04-19T01:42:46.159 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:42:46.159 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:42:46.159 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:46.160 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:42:46.160 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3300", 2024-04-19T01:42:46.160 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:42:46.160 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:46.160 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:46.160 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:42:46.160 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789", 2024-04-19T01:42:46.160 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:42:46.160 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:46.160 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:42:46.160 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:46.160 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789/0", 2024-04-19T01:42:46.160 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6789/0", 2024-04-19T01:42:46.160 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:42:46.161 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:42:46.161 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:42:46.161 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:46.161 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:46.161 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 1, 2024-04-19T01:42:46.161 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "b", 2024-04-19T01:42:46.161 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:42:46.161 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:42:46.161 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:46.161 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:42:46.161 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:3300", 2024-04-19T01:42:46.161 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:42:46.161 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:46.162 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:46.162 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:42:46.162 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789", 2024-04-19T01:42:46.162 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:42:46.162 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:46.162 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:42:46.162 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:46.162 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789/0", 2024-04-19T01:42:46.162 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.71:6789/0", 2024-04-19T01:42:46.162 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:42:46.162 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:42:46.163 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:42:46.163 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:46.163 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:46.163 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 2, 2024-04-19T01:42:46.163 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "c", 2024-04-19T01:42:46.163 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:42:46.163 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:42:46.163 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:46.163 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:42:46.163 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3301", 2024-04-19T01:42:46.163 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:42:46.163 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:46.163 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:46.163 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:42:46.164 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790", 2024-04-19T01:42:46.164 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:42:46.164 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:46.164 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:42:46.164 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:46.164 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790/0", 2024-04-19T01:42:46.164 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6790/0", 2024-04-19T01:42:46.164 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:42:46.164 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:42:46.164 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:42:46.164 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:46.164 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:42:46.165 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:46.165 INFO:teuthology.orchestra.run.smithi062.stdout: "feature_map": { 2024-04-19T01:42:46.165 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": [ 2024-04-19T01:42:46.165 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:46.165 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:42:46.165 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:42:46.165 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:42:46.165 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:46.165 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:42:46.165 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": [ 2024-04-19T01:42:46.165 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:46.166 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:42:46.166 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:42:46.166 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 3 2024-04-19T01:42:46.166 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:46.166 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:42:46.166 INFO:teuthology.orchestra.run.smithi062.stdout: "client": [ 2024-04-19T01:42:46.166 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:46.166 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:42:46.166 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:42:46.166 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 3 2024-04-19T01:42:46.166 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:46.167 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:42:46.167 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:46.167 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false 2024-04-19T01:42:46.167 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-19T01:42:46.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.146+0000 7f2dba22f640 1 -- 172.21.15.62:0/2735091876 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f2da00363d0 msgr2=0x7f2da0038890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:46.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.146+0000 7f2dba22f640 1 --2- 172.21.15.62:0/2735091876 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f2da00363d0 0x7f2da0038890 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2db40762f0 tx=0x7f2dac0a7070 comp rx=0 tx=0).stop 2024-04-19T01:42:46.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.146+0000 7f2dba22f640 10 monclient: shutdown 2024-04-19T01:42:46.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.146+0000 7f2dba22f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:46.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.146+0000 7f2dba22f640 1 -- 172.21.15.62:0/2735091876 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2db40a4bc0 msgr2=0x7f2db4134fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:46.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.146+0000 7f2dba22f640 1 --2- 172.21.15.62:0/2735091876 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2db40a4bc0 0x7f2db4134fb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2da800b580 tx=0x7f2da800ba40 comp rx=0 tx=0).stop 2024-04-19T01:42:46.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.146+0000 7f2db3fff640 1 -- 172.21.15.62:0/2735091876 reap_dead start 2024-04-19T01:42:46.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.146+0000 7f2d9bfff640 1 -- 172.21.15.62:0/2735091876 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 4738+0+0 (unknown 0 0 0) 0x7f2da8048970 con 0x7f2db40a4bc0 2024-04-19T01:42:46.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.146+0000 7f2db3fff640 1 -- 172.21.15.62:0/2735091876 reap_dead start 2024-04-19T01:42:46.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.146+0000 7f2dba22f640 1 -- 172.21.15.62:0/2735091876 shutdown_connections 2024-04-19T01:42:46.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.146+0000 7f2dba22f640 1 -- 172.21.15.62:0/2735091876 >> 172.21.15.62:0/2735091876 conn(0x7f2db40a23d0 msgr2=0x7f2db409ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:46.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.146+0000 7f2dba22f640 1 -- 172.21.15.62:0/2735091876 shutdown_connections 2024-04-19T01:42:46.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.146+0000 7f2dba22f640 1 -- 172.21.15.62:0/2735091876 wait complete. 2024-04-19T01:42:46.170 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-19T01:42:46.170 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-04-19T01:42:46.170 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T01:42:46.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f66aefc3640 10 monclient: get_monmap_and_config 2024-04-19T01:42:46.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f66aefc3640 10 monclient: build_initial_monmap 2024-04-19T01:42:46.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f66aefc3640 10 monclient: monmap: 2024-04-19T01:42:46.292 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:46.292 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:46.292 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:42:46.296127+0000 2024-04-19T01:42:46.292 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:42:46.296127+0000 2024-04-19T01:42:46.292 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:46.292 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:46.292 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:42:46.292 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:42:46.292 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:42:46.293 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:42:46.293 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:42:46.293 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:46.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f66aefc3640 1 Processor -- start 2024-04-19T01:42:46.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f66aefc3640 1 -- start start 2024-04-19T01:42:46.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f66aefc3640 10 monclient: init 2024-04-19T01:42:46.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f66aefc3640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:46.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f66aefc3640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-04-19T01:42:46.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f66aefc3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f66a80a47e0 addr v1:172.21.15.71:6789/0 2024-04-19T01:42:46.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f66aefc3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f66a80a4bc0 0x7f66a80a4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:46.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f66aefc3640 10 monclient(hunting): picked mon.noname-b con 0x7f66a80a4bc0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:46.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f66aefc3640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f66a8139560 0x7f66a813b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:46.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f66aefc3640 10 monclient(hunting): picked mon.noname-a con 0x7f66a8139560 addr v2:172.21.15.62:3300/0 2024-04-19T01:42:46.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f66aefc3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66a809fa60 con 0x7f66a80a47e0 2024-04-19T01:42:46.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f66aefc3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:46.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f66aefc3640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f66a809e690 con 0x7f66a8139560 2024-04-19T01:42:46.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f66aefc3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:46.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f66aefc3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f66a809ec50 con 0x7f66a80a4bc0 2024-04-19T01:42:46.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f66aefc3640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:46.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f66aefc3640 10 monclient(hunting): authenticate will time out at 952.115429s 2024-04-19T01:42:46.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f66ad560640 1 -- >> v2:172.21.15.62:3300/0 conn(0x7f66a8139560 msgr2=0x7f66a813b950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.62:3300/0 2024-04-19T01:42:46.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f66ad560640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f66a8139560 0x7f66a813b950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:42:46.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f669ffff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f66a80a4bc0 0x7f66a80a4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:46.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f669ffff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f66a80a4bc0 0x7f66a80a4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:59204/0 (socket says 172.21.15.62:59204) 2024-04-19T01:42:46.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f669ffff640 1 -- 172.21.15.62:0/4052295870 learned_addr learned my addr 172.21.15.62:0/4052295870 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:46.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f669ffff640 10 monclient(hunting): get_auth_request con 0x7f66a80a4bc0 auth_method 0 2024-04-19T01:42:46.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f669ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:46.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f669ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:46.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f669ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:46.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f669ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:46.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f669ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:46.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f669ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:46.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f669ffff640 10 monclient(hunting): handle_auth_done global_id 4400 payload 274 2024-04-19T01:42:46.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.290+0000 7f669ffff640 1 -- 172.21.15.62:0/4052295870 >> v2:172.21.15.62:3300/0 conn(0x7f66a8139560 msgr2=0x7f66a813b950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:42:46.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.294+0000 7f669ffff640 1 --2- 172.21.15.62:0/4052295870 >> v2:172.21.15.62:3300/0 conn(0x7f66a8139560 0x7f66a813b950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:46.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.294+0000 7f669ffff640 1 -- 172.21.15.62:0/4052295870 >> v1:172.21.15.71:6789/0 conn(0x7f66a80a47e0 legacy=0x7f66a809f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:42:46.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.294+0000 7f669ffff640 10 monclient: _finish_hunting 0 2024-04-19T01:42:46.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.294+0000 7f669ffff640 1 monclient: found mon.noname-b 2024-04-19T01:42:46.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.294+0000 7f669ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:46.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.294+0000 7f669ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:42:46.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.294+0000 7f669ffff640 1 -- 172.21.15.62:0/4052295870 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66a809e970 con 0x7f66a80a4bc0 2024-04-19T01:42:46.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f669ffff640 1 -- 172.21.15.62:0/4052295870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f66a80a4bc0 msgr2=0x7f66a80a4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:46.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f669ffff640 1 --2- 172.21.15.62:0/4052295870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f66a80a4bc0 0x7f66a80a4fe0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6698009870 tx=0x7f669801c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=746ed3ae5e5d3b9e server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:46.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f669f7fe640 1 -- 172.21.15.62:0/4052295870 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66980b1070 con 0x7f66a80a4bc0 2024-04-19T01:42:46.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f669f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:46.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f669f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:42:46.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f669f7fe640 10 monclient: dump: 2024-04-19T01:42:46.301 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:46.301 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:46.301 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:46.301 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:46.301 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:46.301 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:46.301 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:46.301 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:46.301 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:46.301 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:46.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f669f7fe640 10 monclient: _finish_auth 0 2024-04-19T01:42:46.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f669f7fe640 10 monclient: _check_auth_tickets 2024-04-19T01:42:46.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f669f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:46.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f669f7fe640 1 -- 172.21.15.62:0/4052295870 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66980ac410 con 0x7f66a80a4bc0 2024-04-19T01:42:46.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 5 monclient: authenticate success, global_id 4400 2024-04-19T01:42:46.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:42:46.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f669f7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:46.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 10 monclient: get_monmap_and_config success 2024-04-19T01:42:46.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 4 set_mon_vals no callback set 2024-04-19T01:42:46.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f669f7fe640 1 -- 172.21.15.62:0/4052295870 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66980ac7e0 con 0x7f66a80a4bc0 2024-04-19T01:42:46.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f669f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:46.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f669f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:46.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f669f7fe640 10 monclient: dump: 2024-04-19T01:42:46.303 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:46.303 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:46.303 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:46.303 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:46.304 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:46.304 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:46.304 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:46.304 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:46.304 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:46.304 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:46.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 10 monclient: shutdown 2024-04-19T01:42:46.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:46.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 1 -- 172.21.15.62:0/4052295870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f66a80a4bc0 msgr2=0x7f66a80a4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:46.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 1 --2- 172.21.15.62:0/4052295870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f66a80a4bc0 0x7f66a80a4fe0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6698009870 tx=0x7f669801c410 comp rx=0 tx=0).stop 2024-04-19T01:42:46.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 1 -- 172.21.15.62:0/4052295870 shutdown_connections 2024-04-19T01:42:46.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 1 --2- 172.21.15.62:0/4052295870 >> v2:172.21.15.62:3300/0 conn(0x7f66a8139560 0x7f66a813b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:46.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 1 --2- 172.21.15.62:0/4052295870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f66a80a4bc0 0x7f66a80a4fe0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:46.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 1 -- 172.21.15.62:0/4052295870 >> 172.21.15.62:0/4052295870 conn(0x7f66a80a23d0 msgr2=0x7f66a80a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:46.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 1 -- 172.21.15.62:0/4052295870 shutdown_connections 2024-04-19T01:42:46.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 1 -- 172.21.15.62:0/4052295870 wait complete. 2024-04-19T01:42:46.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 10 monclient: build_initial_monmap 2024-04-19T01:42:46.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 10 monclient: monmap: 2024-04-19T01:42:46.305 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:42:46.305 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:42:46.305 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:42:46.305 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:42:46.305 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:42:46.305 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:46.305 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:42:46.305 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:42:46.306 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:42:46.306 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:46.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 1 Processor -- start 2024-04-19T01:42:46.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 1 -- start start 2024-04-19T01:42:46.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 10 monclient: init 2024-04-19T01:42:46.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 10 monclient: _reopen_session rank -1 2024-04-19T01:42:46.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-19T01:42:46.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f66a80a47e0 0x7f66a806d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:46.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 10 monclient(hunting): picked mon.noname-c con 0x7f66a80a47e0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:42:46.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66a80a4bc0 0x7f66a806dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:46.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 10 monclient(hunting): picked mon.noname-a con 0x7f66a80a4bc0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:42:46.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66a8139560 0x7f66a806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:46.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 10 monclient(hunting): picked mon.noname-b con 0x7f66a8139560 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:42:46.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:46.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f66a809beb0 con 0x7f66a80a4bc0 2024-04-19T01:42:46.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:46.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f66a809bd30 con 0x7f66a8139560 2024-04-19T01:42:46.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:46.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f66a809c030 con 0x7f66a80a47e0 2024-04-19T01:42:46.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 10 monclient(hunting): _renew_subs 2024-04-19T01:42:46.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66aefc3640 10 monclient(hunting): authenticate will time out at 952.123637s 2024-04-19T01:42:46.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f669ffff640 1 -- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66a80a4bc0 msgr2=0x7f66a806dea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:3300/0 2024-04-19T01:42:46.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f669ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66a80a4bc0 0x7f66a806dea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:42:46.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66acd5f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f66a80a47e0 0x7f66a806d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:46.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66acd5f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f66a80a47e0 0x7f66a806d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:59208/0 (socket says 172.21.15.62:59208) 2024-04-19T01:42:46.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66acd5f640 1 -- 172.21.15.62:0/2987723834 learned_addr learned my addr 172.21.15.62:0/2987723834 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:42:46.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66acd5f640 10 monclient(hunting): get_auth_request con 0x7f66a80a47e0 auth_method 0 2024-04-19T01:42:46.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66acd5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:46.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66acd5f640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:46.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66acd5f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:46.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66acd5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:46.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66acd5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:46.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.298+0000 7f66acd5f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:46.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f66ad560640 1 --2- 172.21.15.62:0/2987723834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66a8139560 0x7f66a806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:46.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f66acd5f640 10 monclient(hunting): handle_auth_done global_id 4403 payload 824 2024-04-19T01:42:46.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f66acd5f640 1 -- 172.21.15.62:0/2987723834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66a8139560 msgr2=0x7f66a806e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:46.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f66acd5f640 1 --2- 172.21.15.62:0/2987723834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66a8139560 0x7f66a806e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:46.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f66acd5f640 1 -- 172.21.15.62:0/2987723834 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66a80a4bc0 msgr2=0x7f66a806dea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:42:46.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f66acd5f640 1 --2- 172.21.15.62:0/2987723834 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66a80a4bc0 0x7f66a806dea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:42:46.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f66acd5f640 10 monclient: _finish_hunting 0 2024-04-19T01:42:46.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f66acd5f640 1 monclient: found mon.noname-c 2024-04-19T01:42:46.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f66acd5f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:42:46.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f66acd5f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:42:46.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f66acd5f640 1 -- 172.21.15.62:0/2987723834 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66a8070af0 con 0x7f66a80a47e0 2024-04-19T01:42:46.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f66ad560640 10 monclient: get_auth_request con 0x7f66a8139560 auth_method 0 2024-04-19T01:42:46.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f66ad560640 1 --2- 172.21.15.62:0/2987723834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66a8139560 0x7f66a806e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:42:46.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f66acd5f640 1 -- 172.21.15.62:0/2987723834 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f66a80a47e0 msgr2=0x7f66a806d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:46.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f66acd5f640 1 --2- 172.21.15.62:0/2987723834 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f66a80a47e0 0x7f66a806d960 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f66a400c450 tx=0x7f66a400c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:46.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f669d7fa640 1 -- 172.21.15.62:0/2987723834 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a4012a40 con 0x7f66a80a47e0 2024-04-19T01:42:46.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f669d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:46.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f669d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:42:46.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f66acd5f640 1 --2- 172.21.15.62:0/2987723834 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f66a80a47e0 0x7f66a806d960 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f66a400c450 tx=0x7f66a400c910 comp rx=0 tx=0).queue_received will delay after 1.71349e+09 on 0x7f66a4017070 config(0 keys) 2024-04-19T01:42:46.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f669d7fa640 10 monclient: dump: 2024-04-19T01:42:46.317 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:46.317 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:46.318 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:46.318 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:46.318 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:46.318 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:46.318 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:46.318 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:46.318 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:46.318 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:46.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f669d7fa640 10 monclient: _finish_auth 0 2024-04-19T01:42:46.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f669d7fa640 10 monclient: _check_auth_tickets 2024-04-19T01:42:46.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f669d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:42:46.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f66aefc3640 5 monclient: authenticate success, global_id 4403 2024-04-19T01:42:46.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f669d7fa640 1 -- 172.21.15.62:0/2987723834 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66a4017070 con 0x7f66a80a47e0 2024-04-19T01:42:46.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f66aefc3640 10 monclient: _renew_subs 2024-04-19T01:42:46.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f66aefc3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:46.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f66aefc3640 1 -- 172.21.15.62:0/2987723834 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66a8070dd0 con 0x7f66a80a47e0 2024-04-19T01:42:46.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f669d7fa640 10 monclient: handle_config config(0 keys) 2024-04-19T01:42:46.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f66aefc3640 10 monclient: _renew_subs 2024-04-19T01:42:46.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f66aefc3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:42:46.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f66aefc3640 1 -- 172.21.15.62:0/2987723834 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f66a806f630 con 0x7f66a80a47e0 2024-04-19T01:42:46.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f669effd640 4 set_mon_vals no callback set 2024-04-19T01:42:46.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f66aefc3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:46.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f66aefc3640 1 --2- 172.21.15.62:0/2987723834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66a81cb4a0 0x7f66a81cb880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:46.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f66aefc3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:46.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f66aefc3640 1 -- 172.21.15.62:0/2987723834 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f66a80692b0 con 0x7f66a81cb4a0 2024-04-19T01:42:46.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f669d7fa640 1 -- 172.21.15.62:0/2987723834 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a4012e40 con 0x7f66a80a47e0 2024-04-19T01:42:46.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f669d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:46.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f669d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:42:46.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f669d7fa640 10 monclient: dump: 2024-04-19T01:42:46.321 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:46.321 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:46.321 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:46.321 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:46.321 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:46.321 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:46.321 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:46.321 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:46.321 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:46.321 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:46.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f669d7fa640 1 -- 172.21.15.62:0/2987723834 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f66a401c460 con 0x7f66a80a47e0 2024-04-19T01:42:46.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f669ffff640 1 --2- 172.21.15.62:0/2987723834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66a81cb4a0 0x7f66a81cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:46.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f669d7fa640 1 --2- 172.21.15.62:0/2987723834 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f6694036400 0x7f66940388c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:46.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f669ffff640 10 monclient: get_auth_request con 0x7f66a81cb4a0 auth_method 0 2024-04-19T01:42:46.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f669ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:46.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f669ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:46.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f669ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:46.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f669ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:46.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f669ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:46.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.302+0000 7f669ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:46.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.306+0000 7f66ad560640 1 --2- 172.21.15.62:0/2987723834 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f6694036400 0x7f66940388c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:46.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.306+0000 7f66ad560640 10 monclient: get_auth_request con 0x7f6694036400 auth_method 0 2024-04-19T01:42:46.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.306+0000 7f669ffff640 10 monclient(hunting): handle_auth_done global_id 4336 payload 824 2024-04-19T01:42:46.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.306+0000 7f669ffff640 1 -- 172.21.15.62:0/2987723834 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f66a806ed50 con 0x7f66a81cb4a0 2024-04-19T01:42:46.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.306+0000 7f669ffff640 1 -- 172.21.15.62:0/2987723834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66a81cb4a0 msgr2=0x7f66a81cb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:46.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.306+0000 7f669ffff640 1 --2- 172.21.15.62:0/2987723834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66a81cb4a0 0x7f66a81cb880 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f66980ae000 tx=0x7f669801ac50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:46.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.306+0000 7f66ad560640 1 --2- 172.21.15.62:0/2987723834 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f6694036400 0x7f66940388c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f66a80a2de0 tx=0x7f66a0007340 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:46.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.306+0000 7f669d7fa640 1 -- 172.21.15.62:0/2987723834 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66980bb020 con 0x7f66a81cb4a0 2024-04-19T01:42:46.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.306+0000 7f669d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:46.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.306+0000 7f669d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:42:46.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.306+0000 7f669d7fa640 10 monclient: dump: 2024-04-19T01:42:46.324 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:46.324 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:46.324 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:46.324 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:46.324 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:46.324 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:46.325 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:46.325 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:46.325 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:46.325 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:46.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.310+0000 7f669d7fa640 1 -- 172.21.15.62:0/2987723834 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f66980acc10 con 0x7f66a81cb4a0 2024-04-19T01:42:46.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.310+0000 7f669d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:42:46.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.310+0000 7f669d7fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:42:46.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.310+0000 7f669d7fa640 1 -- 172.21.15.62:0/2987723834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66a81cb4a0 msgr2=0x7f66a81cb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:46.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.310+0000 7f669d7fa640 1 --2- 172.21.15.62:0/2987723834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66a81cb4a0 0x7f66a81cb880 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f66980ae000 tx=0x7f669801ac50 comp rx=0 tx=0).stop 2024-04-19T01:42:46.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.310+0000 7f669d7fa640 1 -- 172.21.15.62:0/2987723834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66a81cb4a0 msgr2=0x7f66a81cb880 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:42:46.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.310+0000 7f669d7fa640 1 --2- 172.21.15.62:0/2987723834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66a81cb4a0 0x7f66a81cb880 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f66980ae000 tx=0x7f669801ac50 comp rx=0 tx=0).stop 2024-04-19T01:42:46.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.314+0000 7f66aefc3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:42:46.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.314+0000 7f66aefc3640 1 --2- 172.21.15.62:0/2987723834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66a81cbdc0 0x7f66a8134700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:42:46.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.314+0000 7f66aefc3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:42:46.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.314+0000 7f66aefc3640 1 -- 172.21.15.62:0/2987723834 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f66a809f4c0 con 0x7f66a81cbdc0 2024-04-19T01:42:46.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.314+0000 7f669ffff640 1 --2- 172.21.15.62:0/2987723834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66a81cbdc0 0x7f66a8134700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:42:46.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.314+0000 7f669ffff640 10 monclient: get_auth_request con 0x7f66a81cbdc0 auth_method 0 2024-04-19T01:42:46.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.314+0000 7f669ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:42:46.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.314+0000 7f669ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:42:46.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.314+0000 7f669ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:42:46.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.314+0000 7f669ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:42:46.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.314+0000 7f669ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:42:46.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.314+0000 7f669ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:42:46.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.314+0000 7f669ffff640 10 monclient(hunting): handle_auth_done global_id 4339 payload 824 2024-04-19T01:42:46.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.314+0000 7f669ffff640 1 -- 172.21.15.62:0/2987723834 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f66a806b980 con 0x7f66a81cbdc0 2024-04-19T01:42:46.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.318+0000 7f669ffff640 1 -- 172.21.15.62:0/2987723834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66a81cbdc0 msgr2=0x7f66a8134700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:42:46.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.318+0000 7f669ffff640 1 --2- 172.21.15.62:0/2987723834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66a81cbdc0 0x7f66a8134700 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f669801ccb0 tx=0x7f66980191b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:42:46.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.318+0000 7f669d7fa640 1 -- 172.21.15.62:0/2987723834 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66980bb070 con 0x7f66a81cbdc0 2024-04-19T01:42:46.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.318+0000 7f669d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:42:46.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.318+0000 7f669d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:42:46.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.318+0000 7f669d7fa640 10 monclient: dump: 2024-04-19T01:42:46.329 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:42:46.329 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:42:46.329 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:46.329 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:42:46.329 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:42:46.329 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:42:46.329 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:42:46.329 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:42:46.329 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:42:46.329 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:42:46.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.318+0000 7f669d7fa640 1 -- 172.21.15.62:0/2987723834 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4877 (secure 0 0 0) 0x7f6698003620 con 0x7f66a81cbdc0 2024-04-19T01:42:46.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.318+0000 7f669d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:42:46.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.318+0000 7f669d7fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:42:46.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.318+0000 7f669d7fa640 1 -- 172.21.15.62:0/2987723834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66a81cbdc0 msgr2=0x7f66a8134700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:46.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.318+0000 7f669d7fa640 1 --2- 172.21.15.62:0/2987723834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66a81cbdc0 0x7f66a8134700 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f669801ccb0 tx=0x7f66980191b0 comp rx=0 tx=0).stop 2024-04-19T01:42:46.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.318+0000 7f669d7fa640 1 -- 172.21.15.62:0/2987723834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66a81cbdc0 msgr2=0x7f66a8134700 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:42:46.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.318+0000 7f669d7fa640 1 --2- 172.21.15.62:0/2987723834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66a81cbdc0 0x7f66a8134700 secure :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f669801ccb0 tx=0x7f66980191b0 comp rx=0 tx=0).stop 2024-04-19T01:42:46.330 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-19T01:42:46.330 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "b", 2024-04-19T01:42:46.331 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 1, 2024-04-19T01:42:46.331 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "leader", 2024-04-19T01:42:46.331 INFO:teuthology.orchestra.run.smithi062.stdout: "election_epoch": 10, 2024-04-19T01:42:46.331 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 42495, 2024-04-19T01:42:46.331 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum": [ 2024-04-19T01:42:46.331 INFO:teuthology.orchestra.run.smithi062.stdout: 1, 2024-04-19T01:42:46.331 INFO:teuthology.orchestra.run.smithi062.stdout: 2 2024-04-19T01:42:46.331 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:42:46.331 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_age": 1, 2024-04-19T01:42:46.331 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:42:46.331 INFO:teuthology.orchestra.run.smithi062.stdout: "required_con": "2450521708007866372", 2024-04-19T01:42:46.331 INFO:teuthology.orchestra.run.smithi062.stdout: "required_mon": [ 2024-04-19T01:42:46.332 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:42:46.332 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:42:46.332 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:42:46.332 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:42:46.332 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:42:46.332 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:42:46.332 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:42:46.332 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:42:46.332 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:42:46.332 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:42:46.332 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:42:46.332 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:42:46.332 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:42:46.333 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_mon": [ 2024-04-19T01:42:46.333 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:42:46.333 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:42:46.333 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:42:46.333 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:42:46.333 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:42:46.333 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:42:46.334 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:42:46.334 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:42:46.334 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:42:46.334 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:42:46.334 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:42:46.334 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:42:46.334 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:46.334 INFO:teuthology.orchestra.run.smithi062.stdout: "outside_quorum": [], 2024-04-19T01:42:46.334 INFO:teuthology.orchestra.run.smithi062.stdout: "extra_probe_peers": [], 2024-04-19T01:42:46.335 INFO:teuthology.orchestra.run.smithi062.stdout: "sync_provider": [], 2024-04-19T01:42:46.335 INFO:teuthology.orchestra.run.smithi062.stdout: "monmap": { 2024-04-19T01:42:46.335 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 1, 2024-04-19T01:42:46.335 INFO:teuthology.orchestra.run.smithi062.stdout: "fsid": "6466e4b6-2520-485f-95b3-9401915fbbad", 2024-04-19T01:42:46.335 INFO:teuthology.orchestra.run.smithi062.stdout: "modified": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:42:46.335 INFO:teuthology.orchestra.run.smithi062.stdout: "created": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:42:46.335 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release": 19, 2024-04-19T01:42:46.335 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release_name": "squid", 2024-04-19T01:42:46.335 INFO:teuthology.orchestra.run.smithi062.stdout: "election_strategy": 1, 2024-04-19T01:42:46.335 INFO:teuthology.orchestra.run.smithi062.stdout: "disallowed_leaders: ": "", 2024-04-19T01:42:46.335 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false, 2024-04-19T01:42:46.336 INFO:teuthology.orchestra.run.smithi062.stdout: "tiebreaker_mon": "", 2024-04-19T01:42:46.336 INFO:teuthology.orchestra.run.smithi062.stdout: "removed_ranks: ": "", 2024-04-19T01:42:46.336 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:42:46.336 INFO:teuthology.orchestra.run.smithi062.stdout: "persistent": [ 2024-04-19T01:42:46.336 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:42:46.336 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:42:46.336 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:42:46.336 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:42:46.336 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:42:46.336 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:42:46.337 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:42:46.337 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:42:46.337 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:42:46.337 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:42:46.337 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:42:46.337 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:42:46.337 INFO:teuthology.orchestra.run.smithi062.stdout: "optional": [] 2024-04-19T01:42:46.337 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:46.337 INFO:teuthology.orchestra.run.smithi062.stdout: "mons": [ 2024-04-19T01:42:46.337 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:46.338 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 0, 2024-04-19T01:42:46.338 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "a", 2024-04-19T01:42:46.338 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:42:46.338 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:42:46.338 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:46.338 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:42:46.338 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3300", 2024-04-19T01:42:46.338 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:42:46.338 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:46.339 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:46.339 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:42:46.339 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789", 2024-04-19T01:42:46.339 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:42:46.339 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:46.339 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:42:46.340 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:46.340 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789/0", 2024-04-19T01:42:46.340 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6789/0", 2024-04-19T01:42:46.340 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:42:46.340 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:42:46.340 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:42:46.340 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:46.340 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:46.340 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 1, 2024-04-19T01:42:46.340 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "b", 2024-04-19T01:42:46.341 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:42:46.341 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:42:46.341 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:46.341 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:42:46.341 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:3300", 2024-04-19T01:42:46.341 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:42:46.341 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:46.341 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:46.341 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:42:46.342 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789", 2024-04-19T01:42:46.342 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:42:46.342 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:46.342 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:42:46.342 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:46.342 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789/0", 2024-04-19T01:42:46.342 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.71:6789/0", 2024-04-19T01:42:46.342 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:42:46.342 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:42:46.342 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:42:46.343 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:46.343 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:46.343 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 2, 2024-04-19T01:42:46.343 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "c", 2024-04-19T01:42:46.343 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:42:46.343 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:42:46.343 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:46.343 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:42:46.343 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3301", 2024-04-19T01:42:46.343 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:42:46.343 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:46.343 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:46.343 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:42:46.343 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790", 2024-04-19T01:42:46.343 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:42:46.344 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:46.344 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:42:46.344 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:46.344 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790/0", 2024-04-19T01:42:46.344 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6790/0", 2024-04-19T01:42:46.344 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:42:46.344 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:42:46.344 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:42:46.344 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:46.344 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:42:46.344 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:46.344 INFO:teuthology.orchestra.run.smithi062.stdout: "feature_map": { 2024-04-19T01:42:46.344 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": [ 2024-04-19T01:42:46.344 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:46.344 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:42:46.345 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:42:46.345 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:42:46.345 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:46.345 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:42:46.345 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": [ 2024-04-19T01:42:46.345 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:46.345 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:42:46.345 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:42:46.345 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 3 2024-04-19T01:42:46.345 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:46.345 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:42:46.345 INFO:teuthology.orchestra.run.smithi062.stdout: "client": [ 2024-04-19T01:42:46.345 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:46.345 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:42:46.345 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:42:46.345 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 6 2024-04-19T01:42:46.346 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:46.346 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:42:46.346 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": [ 2024-04-19T01:42:46.346 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:42:46.346 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:42:46.346 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:42:46.346 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:42:46.346 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:42:46.346 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:42:46.346 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:42:46.346 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false 2024-04-19T01:42:46.346 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-19T01:42:46.348 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27508]: starting. 2024-04-19T01:42:46.348 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27508]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-27507 2024-04-19T01:42:46.348 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27508]: created object 0... 2024-04-19T01:42:46.348 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27508]: created object 25... 2024-04-19T01:42:46.348 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27508]: created object 49... 2024-04-19T01:42:46.348 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27508]: finishing. 2024-04-19T01:42:46.348 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27508]: shutting down. 2024-04-19T01:42:46.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.318+0000 7f66aefc3640 1 -- 172.21.15.62:0/2987723834 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f6694036400 msgr2=0x7f66940388c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:46.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.318+0000 7f66aefc3640 1 --2- 172.21.15.62:0/2987723834 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f6694036400 0x7f66940388c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f66a80a2de0 tx=0x7f66a0007340 comp rx=0 tx=0).stop 2024-04-19T01:42:46.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.318+0000 7f66aefc3640 10 monclient: shutdown 2024-04-19T01:42:46.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.318+0000 7f66aefc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:42:46.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.318+0000 7f66aefc3640 1 -- 172.21.15.62:0/2987723834 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f66a80a47e0 msgr2=0x7f66a806d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:42:46.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.318+0000 7f66aefc3640 1 --2- 172.21.15.62:0/2987723834 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f66a80a47e0 0x7f66a806d960 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f66a400c450 tx=0x7f66a400c910 comp rx=0 tx=0).stop 2024-04-19T01:42:46.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.318+0000 7f66ad560640 1 -- 172.21.15.62:0/2987723834 reap_dead start 2024-04-19T01:42:46.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.318+0000 7f669d7fa640 1 -- 172.21.15.62:0/2987723834 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4738+0+0 (unknown 0 0 0) 0x7f66a4049480 con 0x7f66a80a47e0 2024-04-19T01:42:46.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.318+0000 7f66ad560640 1 -- 172.21.15.62:0/2987723834 reap_dead start 2024-04-19T01:42:46.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.318+0000 7f66aefc3640 1 -- 172.21.15.62:0/2987723834 shutdown_connections 2024-04-19T01:42:46.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.318+0000 7f66aefc3640 1 -- 172.21.15.62:0/2987723834 >> 172.21.15.62:0/2987723834 conn(0x7f66a80a23d0 msgr2=0x7f66a8138b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:42:46.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.318+0000 7f66aefc3640 1 -- 172.21.15.62:0/2987723834 shutdown_connections 2024-04-19T01:42:46.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:42:46.318+0000 7f66aefc3640 1 -- 172.21.15.62:0/2987723834 wait complete. 2024-04-19T01:42:46.350 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-19T01:42:46.350 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-04-19T01:42:47.179 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[27509]: starting. 2024-04-19T01:42:47.179 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[27509]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-27507 2024-04-19T01:42:47.179 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[27509]: shutting down. 2024-04-19T01:42:49.816 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:42:49.817 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27544]: starting. 2024-04-19T01:42:49.817 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27544]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-27507 2024-04-19T01:42:49.817 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27544]: created object 0... 2024-04-19T01:42:49.817 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27544]: created object 25... 2024-04-19T01:42:49.817 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27544]: created object 49... 2024-04-19T01:42:49.817 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27544]: finishing. 2024-04-19T01:42:49.817 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27544]: shutting down. 2024-04-19T01:42:49.873 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:42:49.874 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[27546]: starting. 2024-04-19T01:42:49.874 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[27546]: listing objects. 2024-04-19T01:42:49.874 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[27546]: listed object 0... 2024-04-19T01:42:49.874 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[27546]: listed object 25... 2024-04-19T01:42:49.874 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[27546]: saw 50 objects 2024-04-19T01:42:49.874 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[27546]: shutting down. 2024-04-19T01:42:50.610 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:42:50.610 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[27545]: starting. 2024-04-19T01:42:50.610 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[27545]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-27507 2024-04-19T01:42:50.610 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[27545]: shutting down. 2024-04-19T01:42:50.613 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:42:50.613 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:42:50.613 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:42:50.616 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:42:52.850 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27599]: starting. 2024-04-19T01:42:52.850 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27599]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-27598 2024-04-19T01:42:52.850 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27599]: created object 0... 2024-04-19T01:42:52.850 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27599]: created object 25... 2024-04-19T01:42:52.851 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27599]: created object 49... 2024-04-19T01:42:52.851 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27599]: finishing. 2024-04-19T01:42:52.851 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27599]: shutting down. 2024-04-19T01:42:53.629 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[27600]: starting. 2024-04-19T01:42:53.629 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[27600]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-27598 2024-04-19T01:42:53.629 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[27600]: shutting down. 2024-04-19T01:42:55.875 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:42:55.875 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27635]: starting. 2024-04-19T01:42:55.875 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27635]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-27598 2024-04-19T01:42:55.875 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27635]: created object 0... 2024-04-19T01:42:55.875 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27635]: created object 25... 2024-04-19T01:42:55.875 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27635]: created object 49... 2024-04-19T01:42:55.875 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27635]: finishing. 2024-04-19T01:42:55.875 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27635]: shutting down. 2024-04-19T01:42:55.938 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:42:55.938 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[27637]: starting. 2024-04-19T01:42:55.939 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[27637]: listing objects. 2024-04-19T01:42:55.939 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[27637]: listed object 0... 2024-04-19T01:42:55.939 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[27637]: listed object 25... 2024-04-19T01:42:55.939 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[27637]: saw 50 objects 2024-04-19T01:42:55.939 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[27637]: shutting down. 2024-04-19T01:42:56.656 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:42:56.656 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[27636]: starting. 2024-04-19T01:42:56.656 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[27636]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-27598 2024-04-19T01:42:56.656 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[27636]: shutting down. 2024-04-19T01:42:56.663 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:42:56.663 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:42:56.663 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:42:56.663 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:42:58.907 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27690]: starting. 2024-04-19T01:42:58.907 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27690]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-27689 2024-04-19T01:42:58.907 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27690]: created object 0... 2024-04-19T01:42:58.907 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27690]: created object 25... 2024-04-19T01:42:58.907 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27690]: created object 49... 2024-04-19T01:42:58.907 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27690]: finishing. 2024-04-19T01:42:58.907 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27690]: shutting down. 2024-04-19T01:43:00.087 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[27691]: starting. 2024-04-19T01:43:00.087 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[27691]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-27689 2024-04-19T01:43:00.087 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[27691]: shutting down. 2024-04-19T01:43:02.325 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:02.325 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27726]: starting. 2024-04-19T01:43:02.325 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27726]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-27689 2024-04-19T01:43:02.325 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27726]: created object 0... 2024-04-19T01:43:02.326 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27726]: created object 25... 2024-04-19T01:43:02.326 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27726]: created object 49... 2024-04-19T01:43:02.326 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27726]: finishing. 2024-04-19T01:43:02.326 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27726]: shutting down. 2024-04-19T01:43:02.380 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:02.381 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[27728]: starting. 2024-04-19T01:43:02.381 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[27728]: listing objects. 2024-04-19T01:43:02.381 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[27728]: listed object 0... 2024-04-19T01:43:02.381 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[27728]: listed object 25... 2024-04-19T01:43:02.381 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[27728]: saw 50 objects 2024-04-19T01:43:02.381 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[27728]: shutting down. 2024-04-19T01:43:03.104 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:03.104 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[27727]: starting. 2024-04-19T01:43:03.104 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[27727]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-27689 2024-04-19T01:43:03.104 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[27727]: shutting down. 2024-04-19T01:43:03.107 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:03.107 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:03.107 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:43:03.111 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:43:05.408 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27781]: starting. 2024-04-19T01:43:05.408 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27781]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-27780 2024-04-19T01:43:05.408 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27781]: created object 0... 2024-04-19T01:43:05.408 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27781]: created object 25... 2024-04-19T01:43:05.408 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27781]: created object 49... 2024-04-19T01:43:05.408 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27781]: finishing. 2024-04-19T01:43:05.409 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27781]: shutting down. 2024-04-19T01:43:06.186 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[27782]: starting. 2024-04-19T01:43:06.187 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[27782]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-27780 2024-04-19T01:43:06.187 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[27782]: shutting down. 2024-04-19T01:43:08.420 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:08.420 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27817]: starting. 2024-04-19T01:43:08.420 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27817]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-27780 2024-04-19T01:43:08.420 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27817]: created object 0... 2024-04-19T01:43:08.420 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27817]: created object 25... 2024-04-19T01:43:08.420 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27817]: created object 49... 2024-04-19T01:43:08.420 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27817]: finishing. 2024-04-19T01:43:08.421 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27817]: shutting down. 2024-04-19T01:43:08.475 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:08.476 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[27819]: starting. 2024-04-19T01:43:08.476 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[27819]: listing objects. 2024-04-19T01:43:08.476 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[27819]: listed object 0... 2024-04-19T01:43:08.476 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[27819]: listed object 25... 2024-04-19T01:43:08.476 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[27819]: saw 50 objects 2024-04-19T01:43:08.476 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[27819]: shutting down. 2024-04-19T01:43:09.224 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:09.224 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[27818]: starting. 2024-04-19T01:43:09.224 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[27818]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-27780 2024-04-19T01:43:09.224 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[27818]: shutting down. 2024-04-19T01:43:09.227 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:09.227 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:09.227 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:43:09.230 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:43:11.431 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27872]: starting. 2024-04-19T01:43:11.432 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27872]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-27871 2024-04-19T01:43:11.432 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27872]: created object 0... 2024-04-19T01:43:11.432 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27872]: created object 25... 2024-04-19T01:43:11.432 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27872]: created object 49... 2024-04-19T01:43:11.432 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27872]: finishing. 2024-04-19T01:43:11.432 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27872]: shutting down. 2024-04-19T01:43:12.234 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[27873]: starting. 2024-04-19T01:43:12.234 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[27873]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-27871 2024-04-19T01:43:12.234 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[27873]: shutting down. 2024-04-19T01:43:14.294 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:14.295 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27908]: starting. 2024-04-19T01:43:14.295 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27908]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-27871 2024-04-19T01:43:14.295 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27908]: created object 0... 2024-04-19T01:43:14.295 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27908]: created object 25... 2024-04-19T01:43:14.295 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27908]: created object 49... 2024-04-19T01:43:14.295 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27908]: finishing. 2024-04-19T01:43:14.295 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27908]: shutting down. 2024-04-19T01:43:14.353 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:14.353 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[27910]: starting. 2024-04-19T01:43:14.354 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[27910]: listing objects. 2024-04-19T01:43:14.354 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[27910]: listed object 0... 2024-04-19T01:43:14.354 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[27910]: listed object 25... 2024-04-19T01:43:14.354 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[27910]: saw 50 objects 2024-04-19T01:43:14.354 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[27910]: shutting down. 2024-04-19T01:43:15.884 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:15.884 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[27909]: starting. 2024-04-19T01:43:15.885 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[27909]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-27871 2024-04-19T01:43:15.885 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[27909]: shutting down. 2024-04-19T01:43:15.887 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:15.887 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:15.887 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:43:15.890 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:43:18.117 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27963]: starting. 2024-04-19T01:43:18.117 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27963]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-27962 2024-04-19T01:43:18.117 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27963]: created object 0... 2024-04-19T01:43:18.117 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27963]: created object 25... 2024-04-19T01:43:18.117 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27963]: created object 49... 2024-04-19T01:43:18.117 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27963]: finishing. 2024-04-19T01:43:18.118 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[27963]: shutting down. 2024-04-19T01:43:19.920 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[27964]: starting. 2024-04-19T01:43:19.920 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[27964]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-27962 2024-04-19T01:43:19.920 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[27964]: shutting down. 2024-04-19T01:43:22.168 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:22.169 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27999]: starting. 2024-04-19T01:43:22.169 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27999]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-27962 2024-04-19T01:43:22.169 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27999]: created object 0... 2024-04-19T01:43:22.169 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27999]: created object 25... 2024-04-19T01:43:22.169 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27999]: created object 49... 2024-04-19T01:43:22.169 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27999]: finishing. 2024-04-19T01:43:22.169 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[27999]: shutting down. 2024-04-19T01:43:22.228 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:22.228 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28001]: starting. 2024-04-19T01:43:22.228 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28001]: listing objects. 2024-04-19T01:43:22.228 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28001]: listed object 0... 2024-04-19T01:43:22.229 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28001]: listed object 25... 2024-04-19T01:43:22.229 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28001]: saw 50 objects 2024-04-19T01:43:22.229 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28001]: shutting down. 2024-04-19T01:43:22.959 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:22.959 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28000]: starting. 2024-04-19T01:43:22.959 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28000]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-27962 2024-04-19T01:43:22.959 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28000]: shutting down. 2024-04-19T01:43:22.962 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:22.963 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:22.963 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:43:22.966 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:43:25.072 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28054]: starting. 2024-04-19T01:43:25.073 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28054]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-28053 2024-04-19T01:43:25.073 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28054]: created object 0... 2024-04-19T01:43:25.073 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28054]: created object 25... 2024-04-19T01:43:25.073 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28054]: created object 49... 2024-04-19T01:43:25.073 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28054]: finishing. 2024-04-19T01:43:25.073 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28054]: shutting down. 2024-04-19T01:43:25.874 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28055]: starting. 2024-04-19T01:43:25.874 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28055]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-28053 2024-04-19T01:43:25.874 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28055]: shutting down. 2024-04-19T01:43:28.106 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:28.106 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28090]: starting. 2024-04-19T01:43:28.107 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28090]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-28053 2024-04-19T01:43:28.107 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28090]: created object 0... 2024-04-19T01:43:28.107 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28090]: created object 25... 2024-04-19T01:43:28.107 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28090]: created object 49... 2024-04-19T01:43:28.107 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28090]: finishing. 2024-04-19T01:43:28.107 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28090]: shutting down. 2024-04-19T01:43:28.168 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:28.168 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28092]: starting. 2024-04-19T01:43:28.168 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28092]: listing objects. 2024-04-19T01:43:28.168 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28092]: listed object 0... 2024-04-19T01:43:28.168 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28092]: listed object 25... 2024-04-19T01:43:28.168 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28092]: saw 50 objects 2024-04-19T01:43:28.168 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28092]: shutting down. 2024-04-19T01:43:28.906 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:28.906 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28091]: starting. 2024-04-19T01:43:28.906 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28091]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-28053 2024-04-19T01:43:28.906 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28091]: shutting down. 2024-04-19T01:43:28.909 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:28.909 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:28.909 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:43:28.912 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:43:31.425 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28145]: starting. 2024-04-19T01:43:31.425 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28145]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-28144 2024-04-19T01:43:31.426 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28145]: created object 0... 2024-04-19T01:43:31.426 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28145]: created object 25... 2024-04-19T01:43:31.426 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28145]: created object 49... 2024-04-19T01:43:31.426 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28145]: finishing. 2024-04-19T01:43:31.426 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28145]: shutting down. 2024-04-19T01:43:32.096 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28146]: starting. 2024-04-19T01:43:32.096 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28146]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-28144 2024-04-19T01:43:32.096 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28146]: shutting down. 2024-04-19T01:43:34.092 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:34.092 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28181]: starting. 2024-04-19T01:43:34.092 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28181]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-28144 2024-04-19T01:43:34.092 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28181]: created object 0... 2024-04-19T01:43:34.092 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28181]: created object 25... 2024-04-19T01:43:34.092 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28181]: created object 49... 2024-04-19T01:43:34.093 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28181]: finishing. 2024-04-19T01:43:34.093 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28181]: shutting down. 2024-04-19T01:43:34.995 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:34.996 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28183]: starting. 2024-04-19T01:43:34.996 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28183]: listing objects. 2024-04-19T01:43:34.996 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28183]: listed object 0... 2024-04-19T01:43:34.996 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28183]: listed object 25... 2024-04-19T01:43:34.996 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28183]: saw 50 objects 2024-04-19T01:43:34.996 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28183]: shutting down. 2024-04-19T01:43:35.876 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:35.876 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28182]: starting. 2024-04-19T01:43:35.876 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28182]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-28144 2024-04-19T01:43:35.876 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28182]: shutting down. 2024-04-19T01:43:35.879 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:35.879 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:35.879 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:43:35.882 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:43:38.119 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28236]: starting. 2024-04-19T01:43:38.120 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28236]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-28235 2024-04-19T01:43:38.120 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28236]: created object 0... 2024-04-19T01:43:38.120 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28236]: created object 25... 2024-04-19T01:43:38.120 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28236]: created object 49... 2024-04-19T01:43:38.120 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28236]: finishing. 2024-04-19T01:43:38.120 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28236]: shutting down. 2024-04-19T01:43:38.917 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28237]: starting. 2024-04-19T01:43:38.917 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28237]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-28235 2024-04-19T01:43:38.917 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28237]: shutting down. 2024-04-19T01:43:41.138 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:41.138 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28272]: starting. 2024-04-19T01:43:41.138 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28272]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-28235 2024-04-19T01:43:41.138 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28272]: created object 0... 2024-04-19T01:43:41.138 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28272]: created object 25... 2024-04-19T01:43:41.138 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28272]: created object 49... 2024-04-19T01:43:41.138 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28272]: finishing. 2024-04-19T01:43:41.138 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28272]: shutting down. 2024-04-19T01:43:41.198 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:41.198 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28274]: starting. 2024-04-19T01:43:41.198 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28274]: listing objects. 2024-04-19T01:43:41.198 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28274]: listed object 0... 2024-04-19T01:43:41.198 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28274]: listed object 25... 2024-04-19T01:43:41.198 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28274]: saw 50 objects 2024-04-19T01:43:41.199 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28274]: shutting down. 2024-04-19T01:43:41.939 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:41.939 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28273]: starting. 2024-04-19T01:43:41.939 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28273]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-28235 2024-04-19T01:43:41.939 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28273]: shutting down. 2024-04-19T01:43:41.942 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:41.942 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:41.942 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:43:41.945 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:43:44.288 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28327]: starting. 2024-04-19T01:43:44.288 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28327]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-28326 2024-04-19T01:43:44.288 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28327]: created object 0... 2024-04-19T01:43:44.288 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28327]: created object 25... 2024-04-19T01:43:44.288 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28327]: created object 49... 2024-04-19T01:43:44.288 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28327]: finishing. 2024-04-19T01:43:44.288 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28327]: shutting down. 2024-04-19T01:43:44.880 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28328]: starting. 2024-04-19T01:43:44.880 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28328]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-28326 2024-04-19T01:43:44.880 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28328]: shutting down. 2024-04-19T01:43:47.096 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:47.096 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28363]: starting. 2024-04-19T01:43:47.096 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28363]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-28326 2024-04-19T01:43:47.096 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28363]: created object 0... 2024-04-19T01:43:47.096 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28363]: created object 25... 2024-04-19T01:43:47.096 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28363]: created object 49... 2024-04-19T01:43:47.096 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28363]: finishing. 2024-04-19T01:43:47.096 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28363]: shutting down. 2024-04-19T01:43:47.154 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:47.154 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28365]: starting. 2024-04-19T01:43:47.154 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28365]: listing objects. 2024-04-19T01:43:47.154 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28365]: listed object 0... 2024-04-19T01:43:47.154 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28365]: listed object 25... 2024-04-19T01:43:47.154 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28365]: saw 50 objects 2024-04-19T01:43:47.154 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28365]: shutting down. 2024-04-19T01:43:47.888 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:47.888 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28364]: starting. 2024-04-19T01:43:47.888 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28364]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-28326 2024-04-19T01:43:47.888 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28364]: shutting down. 2024-04-19T01:43:47.890 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:47.890 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:47.890 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:43:47.893 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:43:50.148 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28418]: starting. 2024-04-19T01:43:50.149 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28418]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-28417 2024-04-19T01:43:50.149 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28418]: created object 0... 2024-04-19T01:43:50.149 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28418]: created object 25... 2024-04-19T01:43:50.149 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28418]: created object 49... 2024-04-19T01:43:50.149 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28418]: finishing. 2024-04-19T01:43:50.149 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28418]: shutting down. 2024-04-19T01:43:50.928 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28419]: starting. 2024-04-19T01:43:50.929 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28419]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-28417 2024-04-19T01:43:50.929 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28419]: shutting down. 2024-04-19T01:43:53.212 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:53.212 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28456]: starting. 2024-04-19T01:43:53.212 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28456]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-28417 2024-04-19T01:43:53.212 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28456]: created object 0... 2024-04-19T01:43:53.212 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28456]: created object 25... 2024-04-19T01:43:53.212 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28456]: created object 49... 2024-04-19T01:43:53.212 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28456]: finishing. 2024-04-19T01:43:53.213 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28456]: shutting down. 2024-04-19T01:43:53.271 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:53.271 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28458]: starting. 2024-04-19T01:43:53.271 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28458]: listing objects. 2024-04-19T01:43:53.271 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28458]: listed object 0... 2024-04-19T01:43:53.271 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28458]: listed object 25... 2024-04-19T01:43:53.271 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28458]: saw 50 objects 2024-04-19T01:43:53.271 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28458]: shutting down. 2024-04-19T01:43:53.870 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:53.870 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28457]: starting. 2024-04-19T01:43:53.870 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28457]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-28417 2024-04-19T01:43:53.870 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28457]: shutting down. 2024-04-19T01:43:53.873 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:53.873 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:43:53.873 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:43:53.877 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:43:56.220 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28511]: starting. 2024-04-19T01:43:56.454 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28511]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-28510 2024-04-19T01:43:56.454 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28511]: created object 0... 2024-04-19T01:43:56.454 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28511]: created object 25... 2024-04-19T01:43:56.455 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28511]: created object 49... 2024-04-19T01:43:56.455 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28511]: finishing. 2024-04-19T01:43:56.455 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28511]: shutting down. 2024-04-19T01:43:58.028 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28512]: starting. 2024-04-19T01:43:58.029 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28512]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-28510 2024-04-19T01:43:58.029 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28512]: shutting down. 2024-04-19T01:44:00.263 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:44:00.263 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28547]: starting. 2024-04-19T01:44:00.263 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28547]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-28510 2024-04-19T01:44:00.263 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28547]: created object 0... 2024-04-19T01:44:00.263 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28547]: created object 25... 2024-04-19T01:44:00.263 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28547]: created object 49... 2024-04-19T01:44:00.264 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28547]: finishing. 2024-04-19T01:44:00.264 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28547]: shutting down. 2024-04-19T01:44:00.303 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:44:00.303 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28549]: starting. 2024-04-19T01:44:00.303 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28549]: listing objects. 2024-04-19T01:44:00.303 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28549]: listed object 0... 2024-04-19T01:44:00.303 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28549]: listed object 25... 2024-04-19T01:44:00.303 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28549]: saw 50 objects 2024-04-19T01:44:00.304 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28549]: shutting down. 2024-04-19T01:44:01.063 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:44:01.063 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28548]: starting. 2024-04-19T01:44:01.063 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28548]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-28510 2024-04-19T01:44:01.063 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28548]: shutting down. 2024-04-19T01:44:01.066 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:44:01.066 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:44:01.066 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:44:01.068 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:44:03.475 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28602]: starting. 2024-04-19T01:44:03.476 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28602]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-28601 2024-04-19T01:44:03.476 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28602]: created object 0... 2024-04-19T01:44:03.476 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28602]: created object 25... 2024-04-19T01:44:03.476 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28602]: created object 49... 2024-04-19T01:44:03.476 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28602]: finishing. 2024-04-19T01:44:03.476 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28602]: shutting down. 2024-04-19T01:44:05.104 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28603]: starting. 2024-04-19T01:44:05.104 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28603]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-28601 2024-04-19T01:44:05.104 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28603]: shutting down. 2024-04-19T01:44:07.340 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:44:07.340 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28638]: starting. 2024-04-19T01:44:07.340 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28638]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-28601 2024-04-19T01:44:07.340 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28638]: created object 0... 2024-04-19T01:44:07.340 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28638]: created object 25... 2024-04-19T01:44:07.340 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28638]: created object 49... 2024-04-19T01:44:07.341 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28638]: finishing. 2024-04-19T01:44:07.341 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28638]: shutting down. 2024-04-19T01:44:07.398 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:44:07.398 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28640]: starting. 2024-04-19T01:44:07.398 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28640]: listing objects. 2024-04-19T01:44:07.398 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28640]: listed object 0... 2024-04-19T01:44:07.398 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28640]: listed object 25... 2024-04-19T01:44:07.398 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28640]: saw 50 objects 2024-04-19T01:44:07.398 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28640]: shutting down. 2024-04-19T01:44:08.128 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:44:08.128 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28639]: starting. 2024-04-19T01:44:08.128 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28639]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-28601 2024-04-19T01:44:08.128 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28639]: shutting down. 2024-04-19T01:44:08.130 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:44:08.131 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:44:08.131 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:44:08.133 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:44:10.356 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28693]: starting. 2024-04-19T01:44:10.357 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28693]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-28692 2024-04-19T01:44:10.357 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28693]: created object 0... 2024-04-19T01:44:10.357 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28693]: created object 25... 2024-04-19T01:44:10.357 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28693]: created object 49... 2024-04-19T01:44:10.357 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28693]: finishing. 2024-04-19T01:44:10.357 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28693]: shutting down. 2024-04-19T01:44:12.607 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28694]: starting. 2024-04-19T01:44:12.607 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28694]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-28692 2024-04-19T01:44:12.607 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28694]: shutting down. 2024-04-19T01:44:14.080 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:44:14.081 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28729]: starting. 2024-04-19T01:44:14.081 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28729]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-28692 2024-04-19T01:44:14.081 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28729]: created object 0... 2024-04-19T01:44:14.081 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28729]: created object 25... 2024-04-19T01:44:14.081 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28729]: created object 49... 2024-04-19T01:44:14.081 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28729]: finishing. 2024-04-19T01:44:14.081 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28729]: shutting down. 2024-04-19T01:44:14.347 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:44:14.347 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28731]: starting. 2024-04-19T01:44:14.347 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28731]: listing objects. 2024-04-19T01:44:14.347 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28731]: listed object 0... 2024-04-19T01:44:14.347 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28731]: listed object 25... 2024-04-19T01:44:14.347 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28731]: saw 50 objects 2024-04-19T01:44:14.348 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28731]: shutting down. 2024-04-19T01:44:14.890 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:44:14.891 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28730]: starting. 2024-04-19T01:44:14.891 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28730]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-28692 2024-04-19T01:44:14.891 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28730]: shutting down. 2024-04-19T01:44:14.893 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:44:14.893 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:44:14.893 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:44:14.895 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:44:16.351 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-19T01:44:16.351 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-04-19T01:44:16.351 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-19T01:44:16.352 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-19T01:44:16.354 INFO:tasks.ceph.mon.a:Started 2024-04-19T01:44:16.354 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-19T01:44:16.355 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:44:16.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92c498a640 10 monclient: get_monmap_and_config 2024-04-19T01:44:16.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92c498a640 10 monclient: build_initial_monmap 2024-04-19T01:44:16.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92c498a640 10 monclient: monmap: 2024-04-19T01:44:16.478 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:16.478 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:16.478 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:44:16.483183+0000 2024-04-19T01:44:16.478 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:44:16.483183+0000 2024-04-19T01:44:16.478 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:16.478 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:16.478 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:44:16.478 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:44:16.478 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:44:16.479 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:44:16.479 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:44:16.479 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:16.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92c498a640 1 Processor -- start 2024-04-19T01:44:16.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92c498a640 1 -- start start 2024-04-19T01:44:16.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92c498a640 10 monclient: init 2024-04-19T01:44:16.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92c498a640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:16.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92c498a640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-04-19T01:44:16.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92c498a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f92c0137780 addr v1:172.21.15.71:6789/0 2024-04-19T01:44:16.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92c498a640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f92c00a60b0 0x7f92c013c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:16.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92c498a640 10 monclient(hunting): picked mon.noname-c con 0x7f92c00a60b0 addr v2:172.21.15.71:3300/0 2024-04-19T01:44:16.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92c498a640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f92c013cc60 0x7f92c013f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:16.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92c498a640 10 monclient(hunting): picked mon.noname-a con 0x7f92c013cc60 addr v2:172.21.15.62:3300/0 2024-04-19T01:44:16.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92c498a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92c009c2e0 con 0x7f92c0137780 2024-04-19T01:44:16.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92c498a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:16.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92c498a640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f92c009bd20 con 0x7f92c013cc60 2024-04-19T01:44:16.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92c498a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:16.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92c498a640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f92c009c000 con 0x7f92c00a60b0 2024-04-19T01:44:16.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92c498a640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:16.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92c498a640 10 monclient(hunting): authenticate will time out at 1042.302261s 2024-04-19T01:44:16.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92bed9d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f92c013cc60 0x7f92c013f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:16.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92bed9d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f92c013cc60 0x7f92c013f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49296/0 (socket says 172.21.15.62:49296) 2024-04-19T01:44:16.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92bed9d640 1 -- 172.21.15.62:0/1589716979 learned_addr learned my addr 172.21.15.62:0/1589716979 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:44:16.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92bed9d640 10 monclient(hunting): get_auth_request con 0x7f92c013cc60 auth_method 0 2024-04-19T01:44:16.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92bed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:16.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92bed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:16.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92bed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:16.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92bed9d640 1 -- 172.21.15.62:0/1589716979 >> v2:172.21.15.62:3300/0 conn(0x7f92c013cc60 msgr2=0x7f92c013f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:44:16.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92bed9d640 1 -- 172.21.15.62:0/1589716979 >> v2:172.21.15.62:3300/0 conn(0x7f92c013cc60 msgr2=0x7f92c013f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:16.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92bed9d640 1 --2- 172.21.15.62:0/1589716979 >> v2:172.21.15.62:3300/0 conn(0x7f92c013cc60 0x7f92c013f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:16.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92bed9d640 1 --2- 172.21.15.62:0/1589716979 >> v2:172.21.15.62:3300/0 conn(0x7f92c013cc60 0x7f92c013f050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:44:16.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92bdd9b640 1 --2- 172.21.15.62:0/1589716979 >> v2:172.21.15.71:3300/0 conn(0x7f92c00a60b0 0x7f92c013c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:16.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92bdd9b640 10 monclient(hunting): get_auth_request con 0x7f92c00a60b0 auth_method 0 2024-04-19T01:44:16.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92bdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:16.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92bdd9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:16.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:16.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92bdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:16.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92bdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:16.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92bdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:16.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92bdd9b640 10 monclient(hunting): handle_auth_done global_id 4654 payload 274 2024-04-19T01:44:16.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92bdd9b640 1 -- 172.21.15.62:0/1589716979 >> v2:172.21.15.62:3300/0 conn(0x7f92c013cc60 msgr2=0x7f92c013f050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:44:16.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92bdd9b640 1 --2- 172.21.15.62:0/1589716979 >> v2:172.21.15.62:3300/0 conn(0x7f92c013cc60 0x7f92c013f050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:16.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92bdd9b640 1 -- 172.21.15.62:0/1589716979 >> v1:172.21.15.71:6789/0 conn(0x7f92c0137780 legacy=0x7f92c0137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:44:16.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92bdd9b640 10 monclient: _finish_hunting 0 2024-04-19T01:44:16.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92bdd9b640 1 monclient: found mon.noname-c 2024-04-19T01:44:16.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92bdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:16.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92bdd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-04-19T01:44:16.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.481+0000 7f92bdd9b640 1 -- 172.21.15.62:0/1589716979 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92c0138270 con 0x7f92c00a60b0 2024-04-19T01:44:16.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.485+0000 7f92bdd9b640 1 -- 172.21.15.62:0/1589716979 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92c00a60b0 msgr2=0x7f92c013c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:16.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.485+0000 7f92bdd9b640 1 --2- 172.21.15.62:0/1589716979 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92c00a60b0 0x7f92c013c720 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f92c00a74f0 tx=0x7f92b401c1f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23d291d19b7d031a server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:16.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92bd59a640 1 -- 172.21.15.62:0/1589716979 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b40b0070 con 0x7f92c00a60b0 2024-04-19T01:44:16.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92bd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:16.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92bd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:44:16.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92bd59a640 10 monclient: dump: 2024-04-19T01:44:16.486 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:16.486 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:16.487 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:16.487 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:16.487 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:16.487 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:16.487 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:16.487 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:16.487 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:16.487 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:16.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92bd59a640 10 monclient: _finish_auth 0 2024-04-19T01:44:16.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92bd59a640 10 monclient: _check_auth_tickets 2024-04-19T01:44:16.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92bd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:16.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 5 monclient: authenticate success, global_id 4654 2024-04-19T01:44:16.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:44:16.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92bd59a640 1 -- 172.21.15.62:0/1589716979 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92b40ab600 con 0x7f92c00a60b0 2024-04-19T01:44:16.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92bd59a640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:16.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 10 monclient: get_monmap_and_config success 2024-04-19T01:44:16.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 4 set_mon_vals no callback set 2024-04-19T01:44:16.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92bd59a640 1 -- 172.21.15.62:0/1589716979 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b40ab9d0 con 0x7f92c00a60b0 2024-04-19T01:44:16.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92bd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:16.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92bd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:44:16.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92bd59a640 10 monclient: dump: 2024-04-19T01:44:16.488 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:16.488 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:16.488 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:16.488 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:16.488 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:16.488 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:16.488 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:16.488 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:16.488 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:16.488 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:16.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 10 monclient: shutdown 2024-04-19T01:44:16.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:16.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 1 -- 172.21.15.62:0/1589716979 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92c00a60b0 msgr2=0x7f92c013c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:16.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 1 --2- 172.21.15.62:0/1589716979 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92c00a60b0 0x7f92c013c720 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f92c00a74f0 tx=0x7f92b401c1f0 comp rx=0 tx=0).stop 2024-04-19T01:44:16.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 1 -- 172.21.15.62:0/1589716979 shutdown_connections 2024-04-19T01:44:16.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 1 --2- 172.21.15.62:0/1589716979 >> v2:172.21.15.62:3300/0 conn(0x7f92c013cc60 0x7f92c013f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:16.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 1 --2- 172.21.15.62:0/1589716979 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92c00a60b0 0x7f92c013c720 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:16.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 1 -- 172.21.15.62:0/1589716979 >> 172.21.15.62:0/1589716979 conn(0x7f92c00aa1d0 msgr2=0x7f92c00a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:16.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 1 -- 172.21.15.62:0/1589716979 shutdown_connections 2024-04-19T01:44:16.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 1 -- 172.21.15.62:0/1589716979 wait complete. 2024-04-19T01:44:16.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 10 monclient: build_initial_monmap 2024-04-19T01:44:16.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 10 monclient: monmap: 2024-04-19T01:44:16.489 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:16.489 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:44:16.489 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:44:16.489 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:44:16.490 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:16.490 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:16.490 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:44:16.490 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:44:16.490 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:44:16.490 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:16.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 1 Processor -- start 2024-04-19T01:44:16.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 1 -- start start 2024-04-19T01:44:16.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 10 monclient: init 2024-04-19T01:44:16.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:16.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-19T01:44:16.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f92c00a60b0 0x7f92c01cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:16.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 10 monclient(hunting): picked mon.noname-c con 0x7f92c00a60b0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:44:16.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92c0137780 0x7f92c01cfb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:16.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 10 monclient(hunting): picked mon.noname-b con 0x7f92c0137780 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:44:16.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92c013cc60 0x7f92c009f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:16.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 10 monclient(hunting): picked mon.noname-a con 0x7f92c013cc60 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:44:16.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:16.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f92c0142140 con 0x7f92c013cc60 2024-04-19T01:44:16.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:16.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f92c0141fc0 con 0x7f92c0137780 2024-04-19T01:44:16.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:16.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f92c01422c0 con 0x7f92c00a60b0 2024-04-19T01:44:16.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:16.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92c498a640 10 monclient(hunting): authenticate will time out at 1042.310369s 2024-04-19T01:44:16.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92be59c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f92c00a60b0 0x7f92c01cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:16.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92be59c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f92c00a60b0 0x7f92c01cf3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:42858/0 (socket says 172.21.15.62:42858) 2024-04-19T01:44:16.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92be59c640 1 -- 172.21.15.62:0/3446709607 learned_addr learned my addr 172.21.15.62:0/3446709607 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:44:16.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92be59c640 10 monclient(hunting): get_auth_request con 0x7f92c00a60b0 auth_method 0 2024-04-19T01:44:16.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:16.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92bed9d640 1 --2- 172.21.15.62:0/3446709607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92c013cc60 0x7f92c009f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:16.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92be59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:16.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92be59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:16.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92bed9d640 10 monclient(hunting): get_auth_request con 0x7f92c013cc60 auth_method 0 2024-04-19T01:44:16.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92bed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:16.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92bed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:16.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92bed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:16.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92be59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:16.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92be59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:16.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92be59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:16.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92bed9d640 1 -- 172.21.15.62:0/3446709607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92c013cc60 msgr2=0x7f92c009f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-19T01:44:16.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92bed9d640 1 -- 172.21.15.62:0/3446709607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92c013cc60 msgr2=0x7f92c009f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:44:16.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92bed9d640 1 --2- 172.21.15.62:0/3446709607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92c013cc60 0x7f92c009f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:16.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92bed9d640 1 --2- 172.21.15.62:0/3446709607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92c013cc60 0x7f92c009f960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:44:16.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92bdd9b640 1 --2- 172.21.15.62:0/3446709607 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92c0137780 0x7f92c01cfb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:16.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92be59c640 10 monclient(hunting): handle_auth_done global_id 4736 payload 824 2024-04-19T01:44:16.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92be59c640 1 -- 172.21.15.62:0/3446709607 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92c0137780 msgr2=0x7f92c01cfb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:16.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92be59c640 1 --2- 172.21.15.62:0/3446709607 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92c0137780 0x7f92c01cfb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:16.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92be59c640 1 -- 172.21.15.62:0/3446709607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92c013cc60 msgr2=0x7f92c009f960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:44:16.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92be59c640 1 --2- 172.21.15.62:0/3446709607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92c013cc60 0x7f92c009f960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:16.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92be59c640 10 monclient: _finish_hunting 0 2024-04-19T01:44:16.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92be59c640 1 monclient: found mon.noname-c 2024-04-19T01:44:16.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92be59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:16.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92be59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:44:16.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.489+0000 7f92be59c640 1 -- 172.21.15.62:0/3446709607 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92c0120cf0 con 0x7f92c00a60b0 2024-04-19T01:44:16.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.493+0000 7f92be59c640 1 -- 172.21.15.62:0/3446709607 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f92c00a60b0 msgr2=0x7f92c01cf3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:16.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.493+0000 7f92be59c640 1 --2- 172.21.15.62:0/3446709607 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f92c00a60b0 0x7f92c01cf3b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f92b000c450 tx=0x7f92b000c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:16.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.493+0000 7f92aeffd640 1 -- 172.21.15.62:0/3446709607 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b0012a40 con 0x7f92c00a60b0 2024-04-19T01:44:16.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.493+0000 7f92aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:16.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.493+0000 7f92aeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:44:16.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.493+0000 7f92aeffd640 10 monclient: dump: 2024-04-19T01:44:16.497 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:16.497 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:16.497 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:16.497 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:16.497 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:16.497 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:16.497 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:16.497 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:16.497 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:16.497 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:16.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.493+0000 7f92aeffd640 10 monclient: _finish_auth 0 2024-04-19T01:44:16.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.493+0000 7f92aeffd640 10 monclient: _check_auth_tickets 2024-04-19T01:44:16.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.493+0000 7f92aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:16.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.493+0000 7f92aeffd640 1 -- 172.21.15.62:0/3446709607 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92b0017070 con 0x7f92c00a60b0 2024-04-19T01:44:16.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.493+0000 7f92c498a640 5 monclient: authenticate success, global_id 4736 2024-04-19T01:44:16.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.493+0000 7f92c498a640 10 monclient: _renew_subs 2024-04-19T01:44:16.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.493+0000 7f92c498a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:44:16.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.493+0000 7f92c498a640 1 -- 172.21.15.62:0/3446709607 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92c0120a10 con 0x7f92c00a60b0 2024-04-19T01:44:16.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.493+0000 7f92aeffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:16.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.493+0000 7f92aeffd640 1 -- 172.21.15.62:0/3446709607 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b0012e40 con 0x7f92c00a60b0 2024-04-19T01:44:16.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.493+0000 7f92aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:16.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.493+0000 7f92aeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:44:16.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.493+0000 7f92aeffd640 10 monclient: dump: 2024-04-19T01:44:16.498 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:16.498 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:16.498 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:16.498 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:16.498 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:16.499 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:16.499 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:16.499 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:16.499 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:16.499 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:16.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.493+0000 7f92c498a640 10 monclient: _renew_subs 2024-04-19T01:44:16.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.493+0000 7f92c498a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:44:16.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.493+0000 7f92c498a640 1 -- 172.21.15.62:0/3446709607 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f92c00a1d00 con 0x7f92c00a60b0 2024-04-19T01:44:16.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.493+0000 7f92affff640 4 set_mon_vals no callback set 2024-04-19T01:44:16.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.493+0000 7f92aeffd640 1 -- 172.21.15.62:0/3446709607 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f92b001c460 con 0x7f92c00a60b0 2024-04-19T01:44:16.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.493+0000 7f92c498a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:16.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.493+0000 7f92c498a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:16.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.493+0000 7f92c498a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:44:16.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.493+0000 7f92c498a640 1 -- 172.21.15.62:0/3446709607 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92c0068ab0 con 0x7f92c00a60b0 2024-04-19T01:44:16.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.493+0000 7f92aeffd640 1 --2- 172.21.15.62:0/3446709607 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f92a0036420 0x7f92a00388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:16.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.493+0000 7f92aeffd640 1 -- 172.21.15.62:0/3446709607 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(106..106 src has 1..106) ==== 4346+0+0 (secure 0 0 0) 0x7f92b00491a0 con 0x7f92c00a60b0 2024-04-19T01:44:16.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.493+0000 7f92bdd9b640 1 --2- 172.21.15.62:0/3446709607 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f92a0036420 0x7f92a00388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:16.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.493+0000 7f92bdd9b640 10 monclient: get_auth_request con 0x7f92a0036420 auth_method 0 2024-04-19T01:44:16.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.497+0000 7f92bdd9b640 1 --2- 172.21.15.62:0/3446709607 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f92a0036420 0x7f92a00388e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f92b40ad000 tx=0x7f92b4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:16.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.497+0000 7f92aeffd640 1 -- 172.21.15.62:0/3446709607 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f92b001c9e0 con 0x7f92c00a60b0 2024-04-19T01:44:16.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.497+0000 7f92aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:16.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.497+0000 7f92aeffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:44:16.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.725+0000 7f92c498a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:44:16.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.725+0000 7f92c498a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:44:16.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.725+0000 7f92c498a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:44:16.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.725+0000 7f92c498a640 1 -- 172.21.15.62:0/3446709607 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f92c0068710 con 0x7f92c00a60b0 2024-04-19T01:44:16.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.725+0000 7f92aeffd640 1 -- 172.21.15.62:0/3446709607 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f92b0047d50 con 0x7f92c00a60b0 2024-04-19T01:44:16.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.725+0000 7f92aeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:44:16.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.725+0000 7f92aeffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:44:16.723 INFO:teuthology.orchestra.run.smithi062.stdout:{"election_epoch":10,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":91,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","modified":"2024-04-19T01:41:38.497795Z","created":"2024-04-19T01:41:38.497795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:44:16.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.729+0000 7f92c498a640 1 -- 172.21.15.62:0/3446709607 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f92a0036420 msgr2=0x7f92a00388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:16.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.729+0000 7f92c498a640 1 --2- 172.21.15.62:0/3446709607 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f92a0036420 0x7f92a00388e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f92b40ad000 tx=0x7f92b4008040 comp rx=0 tx=0).stop 2024-04-19T01:44:16.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.729+0000 7f92c498a640 10 monclient: shutdown 2024-04-19T01:44:16.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.729+0000 7f92c498a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:16.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.729+0000 7f92c498a640 1 -- 172.21.15.62:0/3446709607 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f92c00a60b0 msgr2=0x7f92c01cf3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:16.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.729+0000 7f92c498a640 1 --2- 172.21.15.62:0/3446709607 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f92c00a60b0 0x7f92c01cf3b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f92b000c450 tx=0x7f92b000c910 comp rx=0 tx=0).stop 2024-04-19T01:44:16.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.729+0000 7f92c498a640 1 -- 172.21.15.62:0/3446709607 shutdown_connections 2024-04-19T01:44:16.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.729+0000 7f92c498a640 1 --2- 172.21.15.62:0/3446709607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92c013cc60 0x7f92c009f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:16.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.729+0000 7f92c498a640 1 --2- 172.21.15.62:0/3446709607 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92c0137780 0x7f92c01cfb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:16.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.729+0000 7f92c498a640 1 --2- 172.21.15.62:0/3446709607 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f92a0036420 0x7f92a00388e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:16.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.729+0000 7f92c498a640 1 --2- 172.21.15.62:0/3446709607 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f92c00a60b0 0x7f92c01cf3b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:16.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.729+0000 7f92c498a640 1 -- 172.21.15.62:0/3446709607 >> 172.21.15.62:0/3446709607 conn(0x7f92c00aa1d0 msgr2=0x7f92c013e100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:16.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.729+0000 7f92c498a640 1 -- 172.21.15.62:0/3446709607 shutdown_connections 2024-04-19T01:44:16.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:16.729+0000 7f92c498a640 1 -- 172.21.15.62:0/3446709607 wait complete. 2024-04-19T01:44:19.742 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:44:19.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f0862ba3640 10 monclient: get_monmap_and_config 2024-04-19T01:44:19.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f0862ba3640 10 monclient: build_initial_monmap 2024-04-19T01:44:19.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f0862ba3640 10 monclient: monmap: 2024-04-19T01:44:19.874 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:19.874 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:19.875 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:44:19.879502+0000 2024-04-19T01:44:19.875 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:44:19.879502+0000 2024-04-19T01:44:19.875 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:19.875 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:19.875 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:44:19.875 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:44:19.875 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:44:19.875 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:44:19.875 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:44:19.875 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:19.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f0862ba3640 1 Processor -- start 2024-04-19T01:44:19.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f0862ba3640 1 -- start start 2024-04-19T01:44:19.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f0862ba3640 10 monclient: init 2024-04-19T01:44:19.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f0862ba3640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:19.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f0862ba3640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-04-19T01:44:19.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f0862ba3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f085c0a47e0 0x7f085c0a4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:19.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f0862ba3640 10 monclient(hunting): picked mon.noname-b con 0x7f085c0a47e0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:44:19.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f0862ba3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f085c0a5100 addr v1:172.21.15.71:6789/0 2024-04-19T01:44:19.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f0862ba3640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f085c1357c0 0x7f085c13d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:19.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f0862ba3640 10 monclient(hunting): picked mon.noname-a con 0x7f085c1357c0 addr v2:172.21.15.62:3300/0 2024-04-19T01:44:19.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f0862ba3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f085c09f7c0 con 0x7f085c0a5100 2024-04-19T01:44:19.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f0862ba3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:19.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f0862ba3640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f085c09bf00 con 0x7f085c1357c0 2024-04-19T01:44:19.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f0862ba3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:19.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f0862ba3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f085c09ee90 con 0x7f085c0a47e0 2024-04-19T01:44:19.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f0861140640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f085c1357c0 0x7f085c13d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:19.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f0862ba3640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:19.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f0862ba3640 10 monclient(hunting): authenticate will time out at 1045.699205s 2024-04-19T01:44:19.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f0861140640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f085c1357c0 0x7f085c13d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49314/0 (socket says 172.21.15.62:49314) 2024-04-19T01:44:19.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f0861140640 1 -- 172.21.15.62:0/2611087440 learned_addr learned my addr 172.21.15.62:0/2611087440 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:44:19.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f0861140640 10 monclient(hunting): get_auth_request con 0x7f085c1357c0 auth_method 0 2024-04-19T01:44:19.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f0861140640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:19.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f0861140640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:19.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f0861140640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:19.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f086093f640 1 --2- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f085c0a47e0 0x7f085c0a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:19.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f086093f640 10 monclient(hunting): get_auth_request con 0x7f085c0a47e0 auth_method 0 2024-04-19T01:44:19.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f086093f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:19.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f086093f640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:19.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f086093f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:19.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f0861140640 1 -- 172.21.15.62:0/2611087440 >> v2:172.21.15.62:3300/0 conn(0x7f085c1357c0 msgr2=0x7f085c13d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:44:19.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f0861140640 1 -- 172.21.15.62:0/2611087440 >> v2:172.21.15.62:3300/0 conn(0x7f085c1357c0 msgr2=0x7f085c13d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:19.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f0861140640 1 --2- 172.21.15.62:0/2611087440 >> v2:172.21.15.62:3300/0 conn(0x7f085c1357c0 0x7f085c13d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:19.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f0861140640 1 --2- 172.21.15.62:0/2611087440 >> v2:172.21.15.62:3300/0 conn(0x7f085c1357c0 0x7f085c13d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:44:19.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f086093f640 1 -- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f085c0a47e0 msgr2=0x7f085c0a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:44:19.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f086093f640 1 -- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f085c0a47e0 msgr2=0x7f085c0a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:19.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f086093f640 1 --2- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f085c0a47e0 0x7f085c0a4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:19.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.877+0000 7f086093f640 1 --2- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f085c0a47e0 0x7f085c0a4bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:44:19.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:19.881+0000 7f085bfff640 1 -- 172.21.15.62:0/2611087440 >> v1:172.21.15.71:6789/0 conn(0x7f085c0a5100 legacy=0x7f085c1350b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:20.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.081+0000 7f0861140640 1 --2- 172.21.15.62:0/2611087440 >> v2:172.21.15.62:3300/0 conn(0x7f085c1357c0 0x7f085c13d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:20.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.081+0000 7f086093f640 1 --2- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f085c0a47e0 0x7f085c0a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:20.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.081+0000 7f0861140640 10 monclient(hunting): get_auth_request con 0x7f085c1357c0 auth_method 0 2024-04-19T01:44:20.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.081+0000 7f0861140640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:20.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.081+0000 7f0861140640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:20.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.081+0000 7f0861140640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:20.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.081+0000 7f086093f640 10 monclient(hunting): get_auth_request con 0x7f085c0a47e0 auth_method 0 2024-04-19T01:44:20.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.081+0000 7f086093f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:20.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.081+0000 7f086093f640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:20.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.081+0000 7f086093f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:20.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.081+0000 7f0861140640 1 -- 172.21.15.62:0/2611087440 >> v2:172.21.15.62:3300/0 conn(0x7f085c1357c0 msgr2=0x7f085c13d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:44:20.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.081+0000 7f0861140640 1 -- 172.21.15.62:0/2611087440 >> v2:172.21.15.62:3300/0 conn(0x7f085c1357c0 msgr2=0x7f085c13d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:20.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.081+0000 7f0861140640 1 --2- 172.21.15.62:0/2611087440 >> v2:172.21.15.62:3300/0 conn(0x7f085c1357c0 0x7f085c13d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:20.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.081+0000 7f086093f640 1 -- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f085c0a47e0 msgr2=0x7f085c0a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:44:20.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.081+0000 7f086093f640 1 -- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f085c0a47e0 msgr2=0x7f085c0a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:20.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.081+0000 7f086093f640 1 --2- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f085c0a47e0 0x7f085c0a4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:20.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.081+0000 7f0861140640 1 --2- 172.21.15.62:0/2611087440 >> v2:172.21.15.62:3300/0 conn(0x7f085c1357c0 0x7f085c13d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:44:20.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.081+0000 7f086093f640 1 --2- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f085c0a47e0 0x7f085c0a4bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:44:20.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.485+0000 7f0861140640 1 --2- 172.21.15.62:0/2611087440 >> v2:172.21.15.62:3300/0 conn(0x7f085c1357c0 0x7f085c13d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:20.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.485+0000 7f086093f640 1 --2- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f085c0a47e0 0x7f085c0a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:20.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.485+0000 7f0861140640 10 monclient(hunting): get_auth_request con 0x7f085c1357c0 auth_method 0 2024-04-19T01:44:20.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.485+0000 7f0861140640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:20.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.485+0000 7f0861140640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:20.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.485+0000 7f0861140640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:20.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.485+0000 7f086093f640 10 monclient(hunting): get_auth_request con 0x7f085c0a47e0 auth_method 0 2024-04-19T01:44:20.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.485+0000 7f086093f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:20.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.485+0000 7f086093f640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:20.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.485+0000 7f086093f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:20.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.485+0000 7f086093f640 1 -- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f085c0a47e0 msgr2=0x7f085c0a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:44:20.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.485+0000 7f086093f640 1 -- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f085c0a47e0 msgr2=0x7f085c0a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:20.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.485+0000 7f086093f640 1 --2- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f085c0a47e0 0x7f085c0a4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:20.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.485+0000 7f0861140640 1 -- 172.21.15.62:0/2611087440 >> v2:172.21.15.62:3300/0 conn(0x7f085c1357c0 msgr2=0x7f085c13d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:44:20.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.485+0000 7f0861140640 1 -- 172.21.15.62:0/2611087440 >> v2:172.21.15.62:3300/0 conn(0x7f085c1357c0 msgr2=0x7f085c13d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:20.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.485+0000 7f086093f640 1 --2- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f085c0a47e0 0x7f085c0a4bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:44:20.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.485+0000 7f0861140640 1 --2- 172.21.15.62:0/2611087440 >> v2:172.21.15.62:3300/0 conn(0x7f085c1357c0 0x7f085c13d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:20.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:20.485+0000 7f0861140640 1 --2- 172.21.15.62:0/2611087440 >> v2:172.21.15.62:3300/0 conn(0x7f085c1357c0 0x7f085c13d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:44:21.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:21.289+0000 7f086093f640 1 --2- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f085c0a47e0 0x7f085c0a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:21.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:21.289+0000 7f0861140640 1 --2- 172.21.15.62:0/2611087440 >> v2:172.21.15.62:3300/0 conn(0x7f085c1357c0 0x7f085c13d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:21.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:21.289+0000 7f086093f640 10 monclient(hunting): get_auth_request con 0x7f085c0a47e0 auth_method 0 2024-04-19T01:44:21.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:21.289+0000 7f086093f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:21.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:21.289+0000 7f086093f640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:21.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:21.289+0000 7f086093f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:21.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:21.289+0000 7f0861140640 10 monclient(hunting): get_auth_request con 0x7f085c1357c0 auth_method 0 2024-04-19T01:44:21.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:21.289+0000 7f0861140640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:21.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:21.289+0000 7f0861140640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:21.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:21.289+0000 7f0861140640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:21.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:21.289+0000 7f0861140640 1 -- 172.21.15.62:0/2611087440 >> v2:172.21.15.62:3300/0 conn(0x7f085c1357c0 msgr2=0x7f085c13d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:44:21.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:21.289+0000 7f086093f640 1 -- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f085c0a47e0 msgr2=0x7f085c0a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:44:21.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:21.289+0000 7f0861140640 1 -- 172.21.15.62:0/2611087440 >> v2:172.21.15.62:3300/0 conn(0x7f085c1357c0 msgr2=0x7f085c13d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:21.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:21.289+0000 7f086093f640 1 -- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f085c0a47e0 msgr2=0x7f085c0a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:21.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:21.289+0000 7f0861140640 1 --2- 172.21.15.62:0/2611087440 >> v2:172.21.15.62:3300/0 conn(0x7f085c1357c0 0x7f085c13d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:21.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:21.289+0000 7f086093f640 1 --2- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f085c0a47e0 0x7f085c0a4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:21.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:21.289+0000 7f0861140640 1 --2- 172.21.15.62:0/2611087440 >> v2:172.21.15.62:3300/0 conn(0x7f085c1357c0 0x7f085c13d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:44:21.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:21.289+0000 7f086093f640 1 --2- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f085c0a47e0 0x7f085c0a4bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:44:22.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.877+0000 7f085a7fc640 10 monclient(hunting): tick 2024-04-19T01:44:22.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.877+0000 7f085a7fc640 10 monclient(hunting): _check_auth_tickets 2024-04-19T01:44:22.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.877+0000 7f085a7fc640 1 monclient(hunting): continuing hunt 2024-04-19T01:44:22.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.877+0000 7f085a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T01:44:22.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.877+0000 7f085a7fc640 1 -- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f085c0a47e0 msgr2=0x7f085c0a4bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:44:22.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.877+0000 7f085a7fc640 1 --2- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f085c0a47e0 0x7f085c0a4bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:22.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.877+0000 7f085a7fc640 1 -- 172.21.15.62:0/2611087440 >> v2:172.21.15.62:3300/0 conn(0x7f085c1357c0 msgr2=0x7f085c13d600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:44:22.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.877+0000 7f085a7fc640 1 --2- 172.21.15.62:0/2611087440 >> v2:172.21.15.62:3300/0 conn(0x7f085c1357c0 0x7f085c13d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:22.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.877+0000 7f085a7fc640 1 -- 172.21.15.62:0/2611087440 >> v1:172.21.15.71:6789/0 conn(0x7f085c0a5100 legacy=0x7f085c1350b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:22.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.877+0000 7f085a7fc640 10 monclient: _add_conns ranks=[0,4] 2024-04-19T01:44:22.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.877+0000 7f085a7fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f083c003730 addr v1:172.21.15.62:6789/0 2024-04-19T01:44:22.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.877+0000 7f085a7fc640 1 --2- 172.21.15.62:0/2611087440 >> v2:172.21.15.71:3300/0 conn(0x7f083c0073e0 0x7f083c0098a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:22.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.877+0000 7f085a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f083c0073e0 addr v2:172.21.15.71:3300/0 2024-04-19T01:44:22.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.877+0000 7f085a7fc640 1 -- 172.21.15.62:0/2611087440 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f085c09c2d0 con 0x7f083c003730 2024-04-19T01:44:22.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.877+0000 7f085a7fc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:22.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.877+0000 7f085a7fc640 1 -- 172.21.15.62:0/2611087440 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f085c09ee90 con 0x7f083c0073e0 2024-04-19T01:44:22.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.877+0000 7f085a7fc640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:22.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.877+0000 7f085bfff640 1 --2- 172.21.15.62:0/2611087440 >> v2:172.21.15.71:3300/0 conn(0x7f083c0073e0 0x7f083c0098a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:22.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.877+0000 7f085bfff640 10 monclient(hunting): get_auth_request con 0x7f083c0073e0 auth_method 0 2024-04-19T01:44:22.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.877+0000 7f085bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:22.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.877+0000 7f085bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:22.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.877+0000 7f085bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:22.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.881+0000 7f085bfff640 1 -- 172.21.15.62:0/2611087440 >> v2:172.21.15.71:3300/0 conn(0x7f083c0073e0 msgr2=0x7f083c0098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:44:22.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.881+0000 7f085bfff640 1 -- 172.21.15.62:0/2611087440 >> v2:172.21.15.71:3300/0 conn(0x7f083c0073e0 msgr2=0x7f083c0098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:22.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.881+0000 7f085bfff640 1 --2- 172.21.15.62:0/2611087440 >> v2:172.21.15.71:3300/0 conn(0x7f083c0073e0 0x7f083c0098a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:22.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.881+0000 7f085bfff640 1 --2- 172.21.15.62:0/2611087440 >> v2:172.21.15.71:3300/0 conn(0x7f083c0073e0 0x7f083c0098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:44:22.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.881+0000 7f086093f640 1 -- 172.21.15.62:0/2611087440 >> v1:172.21.15.62:6789/0 conn(0x7f083c003730 legacy=0x7f083c005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:22.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 1 -- 172.21.15.62:0/2611087440 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2279878375 0 0) 0x7f085c09c2d0 con 0x7f083c003730 2024-04-19T01:44:22.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 10 monclient(hunting): _negotiate 2024-04-19T01:44:22.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:22.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:22.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 10 monclient(hunting): my global_id is 14097 2024-04-19T01:44:22.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 1 -- 172.21.15.62:0/2611087440 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0848003530 con 0x7f083c003730 2024-04-19T01:44:22.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 1 -- 172.21.15.62:0/2611087440 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3367612448 0 0) 0x7f0848003530 con 0x7f083c003730 2024-04-19T01:44:22.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 1 -- 172.21.15.62:0/2611087440 >> v2:172.21.15.71:3300/0 conn(0x7f083c0073e0 msgr2=0x7f083c0098a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:44:22.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 1 --2- 172.21.15.62:0/2611087440 >> v2:172.21.15.71:3300/0 conn(0x7f083c0073e0 0x7f083c0098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:22.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 10 monclient: _finish_hunting 0 2024-04-19T01:44:22.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 1 monclient: found mon.noname-a-legacy 2024-04-19T01:44:22.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:22.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.62:6789/0 2024-04-19T01:44:22.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 1 -- 172.21.15.62:0/2611087440 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f083c00afd0 con 0x7f083c003730 2024-04-19T01:44:22.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 10 monclient: _finish_auth 0 2024-04-19T01:44:22.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 10 monclient: _check_auth_tickets 2024-04-19T01:44:22.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:22.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 1 -- 172.21.15.62:0/2611087440 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1244427781 0 0) 0x7f084400f040 con 0x7f083c003730 2024-04-19T01:44:22.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:22.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-19T01:44:22.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 10 monclient: dump: 2024-04-19T01:44:22.964 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:22.964 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:22.964 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:22.964 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:22.964 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:22.965 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:22.965 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:22.965 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:22.965 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:22.965 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:22.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] but i'm connected to v1:172.21.15.62:6789/0, reconnecting 2024-04-19T01:44:22.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:22.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 1 -- 172.21.15.62:0/2611087440 >> v1:172.21.15.62:6789/0 conn(0x7f083c003730 legacy=0x7f083c005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:22.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-19T01:44:22.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f0861140640 1 -- 172.21.15.62:0/2611087440 reap_dead start 2024-04-19T01:44:22.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 1 --2- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0848003ab0 0x7f0848003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:22.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 10 monclient(hunting): picked mon.a con 0x7f0848003ab0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:44:22.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 1 --2- 172.21.15.62:0/2611087440 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0848004810 0x7f0848004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:22.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 10 monclient(hunting): picked mon.b con 0x7f0848004810 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:44:22.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 1 --2- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0848005420 0x7f084800ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:22.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 10 monclient(hunting): picked mon.c con 0x7f0848005420 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:44:22.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:22.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 1 -- 172.21.15.62:0/2611087440 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f085c09ee90 con 0x7f0848003ab0 2024-04-19T01:44:22.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:22.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 1 -- 172.21.15.62:0/2611087440 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f084800c330 con 0x7f0848004810 2024-04-19T01:44:22.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:22.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 1 -- 172.21.15.62:0/2611087440 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f084800c530 con 0x7f0848005420 2024-04-19T01:44:22.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:22.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f085b7fe640 10 monclient(hunting): _finish_auth 0 2024-04-19T01:44:22.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f086093f640 1 --2- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0848003ab0 0x7f0848003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:22.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f086093f640 10 monclient(hunting): get_auth_request con 0x7f0848003ab0 auth_method 0 2024-04-19T01:44:22.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f086093f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:22.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f086093f640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:22.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.961+0000 7f086093f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:22.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.965+0000 7f0861140640 1 --2- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0848005420 0x7f084800ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:22.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.965+0000 7f0861140640 10 monclient(hunting): get_auth_request con 0x7f0848005420 auth_method 0 2024-04-19T01:44:22.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.965+0000 7f0861140640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:22.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.965+0000 7f0861140640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:22.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.965+0000 7f0861140640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:22.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.965+0000 7f086093f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:22.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.965+0000 7f086093f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:22.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.965+0000 7f086093f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T01:44:22.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.965+0000 7f0861140640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:22.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.965+0000 7f0861140640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:22.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.965+0000 7f0861140640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T01:44:22.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.965+0000 7f086093f640 10 monclient(hunting): handle_auth_done global_id 14097 payload 293 2024-04-19T01:44:22.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.965+0000 7f086093f640 1 -- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0848005420 msgr2=0x7f084800ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:44:22.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.965+0000 7f086093f640 1 --2- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0848005420 0x7f084800ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:22.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.965+0000 7f086093f640 1 -- 172.21.15.62:0/2611087440 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0848004810 msgr2=0x7f0848004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:44:22.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.965+0000 7f086093f640 1 --2- 172.21.15.62:0/2611087440 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0848004810 0x7f0848004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:22.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.965+0000 7f086093f640 10 monclient: _finish_hunting 0 2024-04-19T01:44:22.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.965+0000 7f086093f640 1 monclient: found mon.a 2024-04-19T01:44:22.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.965+0000 7f086093f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:22.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.965+0000 7f086093f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:44:22.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.965+0000 7f086093f640 1 -- 172.21.15.62:0/2611087440 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0848003530 con 0x7f0848003ab0 2024-04-19T01:44:22.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.965+0000 7f086093f640 10 monclient: _finish_auth 0 2024-04-19T01:44:22.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.965+0000 7f086093f640 10 monclient: _check_auth_tickets 2024-04-19T01:44:22.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.965+0000 7f086093f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:22.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.965+0000 7f0862ba3640 5 monclient: authenticate success, global_id 14097 2024-04-19T01:44:22.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.965+0000 7f0862ba3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:44:22.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.973+0000 7f086093f640 1 -- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0848003ab0 msgr2=0x7f0848003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:22.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.973+0000 7f086093f640 1 --2- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0848003ab0 0x7f0848003eb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0844000ee0 tx=0x7f084401d990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44c9bd589e36523d server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:22.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.973+0000 7f085b7fe640 1 -- 172.21.15.62:0/2611087440 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08440ac070 con 0x7f0848003ab0 2024-04-19T01:44:22.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.973+0000 7f085b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:22.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.973+0000 7f085b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:44:22.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.973+0000 7f085b7fe640 10 monclient: dump: 2024-04-19T01:44:22.970 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:22.970 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:22.970 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:22.970 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:22.970 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:22.970 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:22.970 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:22.970 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:22.970 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:22.970 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:22.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.973+0000 7f0862ba3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:44:22.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.973+0000 7f085b7fe640 1 -- 172.21.15.62:0/2611087440 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08440a6840 con 0x7f0848003ab0 2024-04-19T01:44:22.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.973+0000 7f085b7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:22.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.973+0000 7f0862ba3640 10 monclient: get_monmap_and_config success 2024-04-19T01:44:22.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.973+0000 7f0862ba3640 4 set_mon_vals no callback set 2024-04-19T01:44:22.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.973+0000 7f085b7fe640 1 -- 172.21.15.62:0/2611087440 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08440142f0 con 0x7f0848003ab0 2024-04-19T01:44:22.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.973+0000 7f085b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:22.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.973+0000 7f085b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:44:22.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.973+0000 7f085b7fe640 10 monclient: dump: 2024-04-19T01:44:22.971 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:22.971 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:22.971 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:22.971 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:22.971 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:22.972 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:22.972 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:22.972 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:22.972 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:22.972 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:22.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.973+0000 7f0862ba3640 10 monclient: shutdown 2024-04-19T01:44:22.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.973+0000 7f0862ba3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:22.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.973+0000 7f0862ba3640 1 -- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0848003ab0 msgr2=0x7f0848003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:22.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.973+0000 7f0862ba3640 1 --2- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0848003ab0 0x7f0848003eb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0844000ee0 tx=0x7f084401d990 comp rx=0 tx=0).stop 2024-04-19T01:44:22.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.973+0000 7f0862ba3640 1 -- 172.21.15.62:0/2611087440 shutdown_connections 2024-04-19T01:44:22.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.973+0000 7f0862ba3640 1 --2- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0848005420 0x7f084800ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:22.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.973+0000 7f0862ba3640 1 --2- 172.21.15.62:0/2611087440 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0848004810 0x7f0848004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:22.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.973+0000 7f0862ba3640 1 --2- 172.21.15.62:0/2611087440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0848003ab0 0x7f0848003eb0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:22.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.973+0000 7f0862ba3640 1 -- 172.21.15.62:0/2611087440 >> 172.21.15.62:0/2611087440 conn(0x7f085c0a23d0 msgr2=0x7f085c0a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:22.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.973+0000 7f0862ba3640 1 -- 172.21.15.62:0/2611087440 shutdown_connections 2024-04-19T01:44:22.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.973+0000 7f0862ba3640 1 -- 172.21.15.62:0/2611087440 wait complete. 2024-04-19T01:44:22.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f0862ba3640 10 monclient: build_initial_monmap 2024-04-19T01:44:22.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f0862ba3640 10 monclient: monmap: 2024-04-19T01:44:22.974 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:22.974 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:44:22.974 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:44:22.974 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:44:22.974 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:22.974 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:22.974 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:44:22.974 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:44:22.974 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:44:22.974 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:22.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f0862ba3640 1 Processor -- start 2024-04-19T01:44:22.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f0862ba3640 1 -- start start 2024-04-19T01:44:22.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f0862ba3640 10 monclient: init 2024-04-19T01:44:22.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f0862ba3640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:22.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f0862ba3640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-19T01:44:22.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f0862ba3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0848003ab0 0x7f085c1cd590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:22.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f0862ba3640 10 monclient(hunting): picked mon.noname-b con 0x7f0848003ab0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:44:22.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f0862ba3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0848004810 0x7f085c134390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:22.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f0862ba3640 10 monclient(hunting): picked mon.noname-a con 0x7f0848004810 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:44:22.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f0862ba3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0848005420 0x7f085c1348d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:22.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f0862ba3640 10 monclient(hunting): picked mon.noname-c con 0x7f0848005420 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:44:22.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f0862ba3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:22.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f0862ba3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f085c09c8e0 con 0x7f0848004810 2024-04-19T01:44:22.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f0862ba3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:22.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f0862ba3640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f085c09c760 con 0x7f0848003ab0 2024-04-19T01:44:22.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f0862ba3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:22.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f0862ba3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f085c1350b0 con 0x7f0848005420 2024-04-19T01:44:22.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f0862ba3640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:22.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f0862ba3640 10 monclient(hunting): authenticate will time out at 1048.798501s 2024-04-19T01:44:22.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f085bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0848004810 0x7f085c134390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:22.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f0861140640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0848005420 0x7f085c1348d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:22.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f086093f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0848003ab0 0x7f085c1cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:22.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f085bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0848004810 0x7f085c134390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49372/0 (socket says 172.21.15.62:49372) 2024-04-19T01:44:22.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f086093f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0848003ab0 0x7f085c1cd590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.62:37774/0 (socket says 172.21.15.62:37774) 2024-04-19T01:44:22.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f085bfff640 1 -- 172.21.15.62:0/196209647 learned_addr learned my addr 172.21.15.62:0/196209647 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:44:22.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f085bfff640 10 monclient(hunting): get_auth_request con 0x7f0848004810 auth_method 0 2024-04-19T01:44:22.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f085bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:22.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f085bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:22.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f085bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:22.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f0861140640 10 monclient(hunting): get_auth_request con 0x7f0848005420 auth_method 0 2024-04-19T01:44:22.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f0861140640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:22.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f0861140640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:22.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f0861140640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:22.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f086093f640 10 monclient(hunting): get_auth_request con 0x7f0848003ab0 auth_method 0 2024-04-19T01:44:22.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f086093f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:22.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f086093f640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:22.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f086093f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:22.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f086093f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:22.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f086093f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:22.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f086093f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:22.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f0861140640 1 -- 172.21.15.62:0/196209647 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0848005420 msgr2=0x7f085c1348d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-19T01:44:22.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f085bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:22.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f0861140640 1 -- 172.21.15.62:0/196209647 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0848005420 msgr2=0x7f085c1348d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:44:22.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f085bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:22.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f0861140640 1 --2- 172.21.15.62:0/196209647 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0848005420 0x7f085c1348d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:22.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f085bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:22.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f0861140640 1 --2- 172.21.15.62:0/196209647 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0848005420 0x7f085c1348d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:44:22.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f086093f640 10 monclient(hunting): handle_auth_done global_id 4657 payload 824 2024-04-19T01:44:22.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f086093f640 1 -- 172.21.15.62:0/196209647 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0848005420 msgr2=0x7f085c1348d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:44:22.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f086093f640 1 --2- 172.21.15.62:0/196209647 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0848005420 0x7f085c1348d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:22.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f086093f640 1 -- 172.21.15.62:0/196209647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0848004810 msgr2=0x7f085c134390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:22.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f086093f640 1 --2- 172.21.15.62:0/196209647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0848004810 0x7f085c134390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:22.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f086093f640 10 monclient: _finish_hunting 0 2024-04-19T01:44:22.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f086093f640 1 monclient: found mon.noname-b 2024-04-19T01:44:22.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f086093f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:22.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f086093f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.71:3300/0 2024-04-19T01:44:22.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f086093f640 1 -- 172.21.15.62:0/196209647 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f085c121a70 con 0x7f0848003ab0 2024-04-19T01:44:22.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.977+0000 7f085bfff640 1 --2- 172.21.15.62:0/196209647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0848004810 0x7f085c134390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:44:22.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.981+0000 7f086093f640 1 -- 172.21.15.62:0/196209647 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0848003ab0 msgr2=0x7f085c1cd590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:22.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.981+0000 7f086093f640 1 --2- 172.21.15.62:0/196209647 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0848003ab0 0x7f085c1cd590 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f084800c300 tx=0x7f084401a830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:22.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.981+0000 7f08597fa640 1 -- 172.21.15.62:0/196209647 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08440ac040 con 0x7f0848003ab0 2024-04-19T01:44:22.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.981+0000 7f08597fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:22.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.981+0000 7f08597fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:44:22.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.981+0000 7f08597fa640 10 monclient: dump: 2024-04-19T01:44:22.984 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:22.984 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:22.984 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:22.984 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:22.984 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:22.984 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:22.984 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:22.984 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:22.985 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:22.985 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:22.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.981+0000 7f08597fa640 10 monclient: _finish_auth 0 2024-04-19T01:44:22.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.981+0000 7f08597fa640 10 monclient: _check_auth_tickets 2024-04-19T01:44:22.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.981+0000 7f08597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:22.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.981+0000 7f0862ba3640 5 monclient: authenticate success, global_id 4657 2024-04-19T01:44:22.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.981+0000 7f0862ba3640 10 monclient: _renew_subs 2024-04-19T01:44:22.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.981+0000 7f0862ba3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:44:22.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.981+0000 7f0862ba3640 1 -- 172.21.15.62:0/196209647 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f085c120550 con 0x7f0848003ab0 2024-04-19T01:44:22.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.981+0000 7f08597fa640 1 -- 172.21.15.62:0/196209647 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f084400e6a0 con 0x7f0848003ab0 2024-04-19T01:44:22.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.981+0000 7f08597fa640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:22.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.981+0000 7f08597fa640 1 -- 172.21.15.62:0/196209647 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08440ba610 con 0x7f0848003ab0 2024-04-19T01:44:22.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.981+0000 7f0862ba3640 10 monclient: _renew_subs 2024-04-19T01:44:22.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.981+0000 7f0862ba3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:44:22.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.981+0000 7f0862ba3640 1 -- 172.21.15.62:0/196209647 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f085c071c10 con 0x7f0848003ab0 2024-04-19T01:44:22.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.981+0000 7f08597fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:22.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.981+0000 7f08597fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:44:22.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.981+0000 7f085affd640 4 set_mon_vals no callback set 2024-04-19T01:44:22.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.981+0000 7f08597fa640 10 monclient: dump: 2024-04-19T01:44:22.986 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:22.986 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:22.986 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:22.986 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:22.986 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:22.986 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:22.986 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:22.986 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:22.986 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:22.986 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:22.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.981+0000 7f0862ba3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:22.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.981+0000 7f0862ba3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:22.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.981+0000 7f0862ba3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:44:22.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.981+0000 7f0862ba3640 1 -- 172.21.15.62:0/196209647 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f085c0692b0 con 0x7f0848003ab0 2024-04-19T01:44:22.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.981+0000 7f08597fa640 1 -- 172.21.15.62:0/196209647 <== mon.1 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f08440b0400 con 0x7f0848003ab0 2024-04-19T01:44:22.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.981+0000 7f08597fa640 1 --2- 172.21.15.62:0/196209647 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f083c036dd0 0x7f083c039290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:22.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.985+0000 7f08597fa640 1 -- 172.21.15.62:0/196209647 <== mon.1 v2:172.21.15.71:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 4738+0+0 (secure 0 0 0) 0x7f08440e3bb0 con 0x7f0848003ab0 2024-04-19T01:44:22.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.985+0000 7f085bfff640 1 --2- 172.21.15.62:0/196209647 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f083c036dd0 0x7f083c039290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:22.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.985+0000 7f085bfff640 10 monclient: get_auth_request con 0x7f083c036dd0 auth_method 0 2024-04-19T01:44:22.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.985+0000 7f085bfff640 1 --2- 172.21.15.62:0/196209647 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f083c036dd0 0x7f083c039290 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f085c0a41c0 tx=0x7f084c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:22.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.997+0000 7f08597fa640 1 -- 172.21.15.62:0/196209647 <== mon.1 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f08440177a0 con 0x7f0848003ab0 2024-04-19T01:44:22.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.997+0000 7f08597fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:22.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:22.997+0000 7f08597fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:44:23.195 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28784]: starting. 2024-04-19T01:44:23.195 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28784]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-28783 2024-04-19T01:44:23.195 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28784]: created object 0... 2024-04-19T01:44:23.195 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28784]: created object 25... 2024-04-19T01:44:23.195 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28784]: created object 49... 2024-04-19T01:44:23.195 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28784]: finishing. 2024-04-19T01:44:23.195 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28784]: shutting down. 2024-04-19T01:44:23.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:23.261+0000 7f0862ba3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:44:23.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:23.261+0000 7f0862ba3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:44:23.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:23.261+0000 7f0862ba3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:44:23.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:23.261+0000 7f0862ba3640 1 -- 172.21.15.62:0/196209647 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f085c120290 con 0x7f0848003ab0 2024-04-19T01:44:23.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:23.261+0000 7f08597fa640 1 -- 172.21.15.62:0/196209647 <== mon.1 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f08440c2db0 con 0x7f0848003ab0 2024-04-19T01:44:23.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:23.261+0000 7f08597fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:44:23.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:23.261+0000 7f08597fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:44:23.259 INFO:teuthology.orchestra.run.smithi062.stdout:{"election_epoch":14,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","modified":"2024-04-19T01:41:38.497795Z","created":"2024-04-19T01:41:38.497795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:44:23.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:23.265+0000 7f0862ba3640 1 -- 172.21.15.62:0/196209647 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f083c036dd0 msgr2=0x7f083c039290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:23.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:23.265+0000 7f0862ba3640 1 --2- 172.21.15.62:0/196209647 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f083c036dd0 0x7f083c039290 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f085c0a41c0 tx=0x7f084c009040 comp rx=0 tx=0).stop 2024-04-19T01:44:23.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:23.265+0000 7f0862ba3640 10 monclient: shutdown 2024-04-19T01:44:23.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:23.265+0000 7f0862ba3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:23.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:23.265+0000 7f0862ba3640 1 -- 172.21.15.62:0/196209647 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0848003ab0 msgr2=0x7f085c1cd590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:23.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:23.265+0000 7f0862ba3640 1 --2- 172.21.15.62:0/196209647 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0848003ab0 0x7f085c1cd590 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f084800c300 tx=0x7f084401a830 comp rx=0 tx=0).stop 2024-04-19T01:44:23.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:23.265+0000 7f0862ba3640 1 -- 172.21.15.62:0/196209647 shutdown_connections 2024-04-19T01:44:23.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:23.265+0000 7f0862ba3640 1 --2- 172.21.15.62:0/196209647 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f083c036dd0 0x7f083c039290 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:23.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:23.265+0000 7f0862ba3640 1 --2- 172.21.15.62:0/196209647 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0848005420 0x7f085c1348d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:23.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:23.265+0000 7f0862ba3640 1 --2- 172.21.15.62:0/196209647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0848004810 0x7f085c134390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:23.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:23.265+0000 7f0862ba3640 1 --2- 172.21.15.62:0/196209647 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0848003ab0 0x7f085c1cd590 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:23.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:23.265+0000 7f0862ba3640 1 -- 172.21.15.62:0/196209647 >> 172.21.15.62:0/196209647 conn(0x7f085c0a23d0 msgr2=0x7f085c09bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:23.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:23.265+0000 7f0862ba3640 1 -- 172.21.15.62:0/196209647 shutdown_connections 2024-04-19T01:44:23.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:23.265+0000 7f0862ba3640 1 -- 172.21.15.62:0/196209647 wait complete. 2024-04-19T01:44:26.279 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:44:26.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd80e21b640 10 monclient: get_monmap_and_config 2024-04-19T01:44:26.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd80e21b640 10 monclient: build_initial_monmap 2024-04-19T01:44:26.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd80e21b640 10 monclient: monmap: 2024-04-19T01:44:26.415 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:26.415 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:26.415 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:44:26.419659+0000 2024-04-19T01:44:26.415 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:44:26.419659+0000 2024-04-19T01:44:26.415 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:26.415 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:26.415 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:44:26.415 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:44:26.415 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:44:26.416 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:44:26.416 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:44:26.416 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:26.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd80e21b640 1 Processor -- start 2024-04-19T01:44:26.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd80e21b640 1 -- start start 2024-04-19T01:44:26.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd80e21b640 10 monclient: init 2024-04-19T01:44:26.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd80e21b640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:26.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd80e21b640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-04-19T01:44:26.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd80e21b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd8080a55e0 0x7fd8080a59c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:26.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd80e21b640 10 monclient(hunting): picked mon.noname-c con 0x7fd8080a55e0 addr v2:172.21.15.71:3300/0 2024-04-19T01:44:26.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd80e21b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd8080a5f00 addr v1:172.21.15.71:6789/0 2024-04-19T01:44:26.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd80e21b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd80813c270 0x7fd808143e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:26.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd80e21b640 10 monclient(hunting): picked mon.noname-a con 0x7fd80813c270 addr v2:172.21.15.62:3300/0 2024-04-19T01:44:26.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd80e21b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd80806ffb0 con 0x7fd8080a5f00 2024-04-19T01:44:26.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd80e21b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:26.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd80e21b640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fd80806f9f0 con 0x7fd80813c270 2024-04-19T01:44:26.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd80e21b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:26.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd80e21b640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fd80806e020 con 0x7fd8080a55e0 2024-04-19T01:44:26.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd80e21b640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:26.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd80e21b640 10 monclient(hunting): authenticate will time out at 1052.239334s 2024-04-19T01:44:26.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd807fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd80813c270 0x7fd808143e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:26.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd807fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd80813c270 0x7fd808143e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36612/0 (socket says 172.21.15.62:36612) 2024-04-19T01:44:26.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd807fff640 1 -- 172.21.15.62:0/1216443517 learned_addr learned my addr 172.21.15.62:0/1216443517 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:44:26.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd807fff640 10 monclient(hunting): get_auth_request con 0x7fd80813c270 auth_method 0 2024-04-19T01:44:26.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd807fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:26.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd807fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:26.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd807fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:26.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd807fff640 1 -- 172.21.15.62:0/1216443517 >> v2:172.21.15.62:3300/0 conn(0x7fd80813c270 msgr2=0x7fd808143e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:44:26.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd807fff640 1 -- 172.21.15.62:0/1216443517 >> v2:172.21.15.62:3300/0 conn(0x7fd80813c270 msgr2=0x7fd808143e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:26.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd807fff640 1 --2- 172.21.15.62:0/1216443517 >> v2:172.21.15.62:3300/0 conn(0x7fd80813c270 0x7fd808143e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:26.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd807fff640 1 --2- 172.21.15.62:0/1216443517 >> v2:172.21.15.62:3300/0 conn(0x7fd80813c270 0x7fd808143e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:44:26.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd8077fe640 1 --2- 172.21.15.62:0/1216443517 >> v2:172.21.15.71:3300/0 conn(0x7fd8080a55e0 0x7fd8080a59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:26.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd8077fe640 10 monclient(hunting): get_auth_request con 0x7fd8080a55e0 auth_method 0 2024-04-19T01:44:26.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd8077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:26.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd8077fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:26.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.417+0000 7fd8077fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:26.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.421+0000 7fd8077fe640 1 -- 172.21.15.62:0/1216443517 >> v2:172.21.15.71:3300/0 conn(0x7fd8080a55e0 msgr2=0x7fd8080a59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:44:26.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.421+0000 7fd8077fe640 1 -- 172.21.15.62:0/1216443517 >> v2:172.21.15.71:3300/0 conn(0x7fd8080a55e0 msgr2=0x7fd8080a59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:26.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.421+0000 7fd8077fe640 1 --2- 172.21.15.62:0/1216443517 >> v2:172.21.15.71:3300/0 conn(0x7fd8080a55e0 0x7fd8080a59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:26.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.421+0000 7fd8077fe640 1 --2- 172.21.15.62:0/1216443517 >> v2:172.21.15.71:3300/0 conn(0x7fd8080a55e0 0x7fd8080a59c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:44:26.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.421+0000 7fd806ffd640 1 -- 172.21.15.62:0/1216443517 >> v1:172.21.15.71:6789/0 conn(0x7fd8080a5f00 legacy=0x7fd80813bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:26.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.621+0000 7fd807fff640 1 --2- 172.21.15.62:0/1216443517 >> v2:172.21.15.62:3300/0 conn(0x7fd80813c270 0x7fd808143e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:26.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.621+0000 7fd807fff640 10 monclient(hunting): get_auth_request con 0x7fd80813c270 auth_method 0 2024-04-19T01:44:26.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.621+0000 7fd807fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:26.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.621+0000 7fd807fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:26.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.621+0000 7fd807fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:26.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.621+0000 7fd807fff640 1 -- 172.21.15.62:0/1216443517 >> v2:172.21.15.62:3300/0 conn(0x7fd80813c270 msgr2=0x7fd808143e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:44:26.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.621+0000 7fd807fff640 1 -- 172.21.15.62:0/1216443517 >> v2:172.21.15.62:3300/0 conn(0x7fd80813c270 msgr2=0x7fd808143e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:26.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.621+0000 7fd807fff640 1 --2- 172.21.15.62:0/1216443517 >> v2:172.21.15.62:3300/0 conn(0x7fd80813c270 0x7fd808143e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:26.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.621+0000 7fd807fff640 1 --2- 172.21.15.62:0/1216443517 >> v2:172.21.15.62:3300/0 conn(0x7fd80813c270 0x7fd808143e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:44:26.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.625+0000 7fd8077fe640 1 --2- 172.21.15.62:0/1216443517 >> v2:172.21.15.71:3300/0 conn(0x7fd8080a55e0 0x7fd8080a59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:26.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.625+0000 7fd8077fe640 10 monclient(hunting): get_auth_request con 0x7fd8080a55e0 auth_method 0 2024-04-19T01:44:26.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.625+0000 7fd8077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:26.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.625+0000 7fd8077fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:26.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.625+0000 7fd8077fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:26.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.625+0000 7fd8077fe640 1 -- 172.21.15.62:0/1216443517 >> v2:172.21.15.71:3300/0 conn(0x7fd8080a55e0 msgr2=0x7fd8080a59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:44:26.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.625+0000 7fd8077fe640 1 -- 172.21.15.62:0/1216443517 >> v2:172.21.15.71:3300/0 conn(0x7fd8080a55e0 msgr2=0x7fd8080a59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:26.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.625+0000 7fd8077fe640 1 --2- 172.21.15.62:0/1216443517 >> v2:172.21.15.71:3300/0 conn(0x7fd8080a55e0 0x7fd8080a59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:26.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:26.625+0000 7fd8077fe640 1 --2- 172.21.15.62:0/1216443517 >> v2:172.21.15.71:3300/0 conn(0x7fd8080a55e0 0x7fd8080a59c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:44:27.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.025+0000 7fd807fff640 1 --2- 172.21.15.62:0/1216443517 >> v2:172.21.15.62:3300/0 conn(0x7fd80813c270 0x7fd808143e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:27.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.025+0000 7fd807fff640 10 monclient(hunting): get_auth_request con 0x7fd80813c270 auth_method 0 2024-04-19T01:44:27.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.025+0000 7fd807fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:27.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.025+0000 7fd807fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:27.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.025+0000 7fd807fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:27.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.025+0000 7fd807fff640 1 -- 172.21.15.62:0/1216443517 >> v2:172.21.15.62:3300/0 conn(0x7fd80813c270 msgr2=0x7fd808143e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:44:27.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.025+0000 7fd807fff640 1 -- 172.21.15.62:0/1216443517 >> v2:172.21.15.62:3300/0 conn(0x7fd80813c270 msgr2=0x7fd808143e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:27.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.025+0000 7fd807fff640 1 --2- 172.21.15.62:0/1216443517 >> v2:172.21.15.62:3300/0 conn(0x7fd80813c270 0x7fd808143e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:27.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.025+0000 7fd807fff640 1 --2- 172.21.15.62:0/1216443517 >> v2:172.21.15.62:3300/0 conn(0x7fd80813c270 0x7fd808143e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:44:27.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.029+0000 7fd8077fe640 1 --2- 172.21.15.62:0/1216443517 >> v2:172.21.15.71:3300/0 conn(0x7fd8080a55e0 0x7fd8080a59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:27.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.029+0000 7fd8077fe640 10 monclient(hunting): get_auth_request con 0x7fd8080a55e0 auth_method 0 2024-04-19T01:44:27.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.029+0000 7fd8077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:27.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.029+0000 7fd8077fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:27.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.029+0000 7fd8077fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:27.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.029+0000 7fd8077fe640 1 -- 172.21.15.62:0/1216443517 >> v2:172.21.15.71:3300/0 conn(0x7fd8080a55e0 msgr2=0x7fd8080a59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:44:27.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.029+0000 7fd8077fe640 1 -- 172.21.15.62:0/1216443517 >> v2:172.21.15.71:3300/0 conn(0x7fd8080a55e0 msgr2=0x7fd8080a59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:27.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.029+0000 7fd8077fe640 1 --2- 172.21.15.62:0/1216443517 >> v2:172.21.15.71:3300/0 conn(0x7fd8080a55e0 0x7fd8080a59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:27.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.029+0000 7fd8077fe640 1 --2- 172.21.15.62:0/1216443517 >> v2:172.21.15.71:3300/0 conn(0x7fd8080a55e0 0x7fd8080a59c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:44:27.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.829+0000 7fd807fff640 1 --2- 172.21.15.62:0/1216443517 >> v2:172.21.15.62:3300/0 conn(0x7fd80813c270 0x7fd808143e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:27.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.829+0000 7fd807fff640 10 monclient(hunting): get_auth_request con 0x7fd80813c270 auth_method 0 2024-04-19T01:44:27.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.829+0000 7fd807fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:27.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.829+0000 7fd807fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:27.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.829+0000 7fd807fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:27.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.829+0000 7fd807fff640 1 -- 172.21.15.62:0/1216443517 >> v2:172.21.15.62:3300/0 conn(0x7fd80813c270 msgr2=0x7fd808143e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:44:27.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.829+0000 7fd807fff640 1 -- 172.21.15.62:0/1216443517 >> v2:172.21.15.62:3300/0 conn(0x7fd80813c270 msgr2=0x7fd808143e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:27.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.829+0000 7fd807fff640 1 --2- 172.21.15.62:0/1216443517 >> v2:172.21.15.62:3300/0 conn(0x7fd80813c270 0x7fd808143e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:27.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.829+0000 7fd807fff640 1 --2- 172.21.15.62:0/1216443517 >> v2:172.21.15.62:3300/0 conn(0x7fd80813c270 0x7fd808143e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:44:27.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.833+0000 7fd8077fe640 1 --2- 172.21.15.62:0/1216443517 >> v2:172.21.15.71:3300/0 conn(0x7fd8080a55e0 0x7fd8080a59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:27.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.833+0000 7fd8077fe640 10 monclient(hunting): get_auth_request con 0x7fd8080a55e0 auth_method 0 2024-04-19T01:44:27.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.833+0000 7fd8077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:27.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.833+0000 7fd8077fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:27.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.833+0000 7fd8077fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:27.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.833+0000 7fd8077fe640 1 -- 172.21.15.62:0/1216443517 >> v2:172.21.15.71:3300/0 conn(0x7fd8080a55e0 msgr2=0x7fd8080a59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:44:27.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.833+0000 7fd8077fe640 1 -- 172.21.15.62:0/1216443517 >> v2:172.21.15.71:3300/0 conn(0x7fd8080a55e0 msgr2=0x7fd8080a59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:27.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.833+0000 7fd8077fe640 1 --2- 172.21.15.62:0/1216443517 >> v2:172.21.15.71:3300/0 conn(0x7fd8080a55e0 0x7fd8080a59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:27.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:27.833+0000 7fd8077fe640 1 --2- 172.21.15.62:0/1216443517 >> v2:172.21.15.71:3300/0 conn(0x7fd8080a55e0 0x7fd8080a59c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:44:28.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 1 -- 172.21.15.62:0/1216443517 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 511357980 0 0) 0x7fd80806ffb0 con 0x7fd8080a5f00 2024-04-19T01:44:28.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 10 monclient(hunting): _negotiate 2024-04-19T01:44:28.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:28.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:28.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 10 monclient(hunting): my global_id is 4660 2024-04-19T01:44:28.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 1 -- 172.21.15.62:0/1216443517 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7f0003530 con 0x7fd8080a5f00 2024-04-19T01:44:28.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 1 -- 172.21.15.62:0/1216443517 <== mon.1 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3991132903 0 0) 0x7fd7f0003530 con 0x7fd8080a5f00 2024-04-19T01:44:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 1 -- 172.21.15.62:0/1216443517 >> v2:172.21.15.71:3300/0 conn(0x7fd8080a55e0 msgr2=0x7fd8080a59c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:44:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 1 --2- 172.21.15.62:0/1216443517 >> v2:172.21.15.71:3300/0 conn(0x7fd8080a55e0 0x7fd8080a59c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 1 -- 172.21.15.62:0/1216443517 >> v2:172.21.15.62:3300/0 conn(0x7fd80813c270 msgr2=0x7fd808143e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:44:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 1 --2- 172.21.15.62:0/1216443517 >> v2:172.21.15.62:3300/0 conn(0x7fd80813c270 0x7fd808143e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 10 monclient: _finish_hunting 0 2024-04-19T01:44:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 1 monclient: found mon.noname-c-legacy 2024-04-19T01:44:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.71:6789/0 2024-04-19T01:44:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 1 -- 172.21.15.62:0/1216443517 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd80806e460 con 0x7fd8080a5f00 2024-04-19T01:44:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 10 monclient: _finish_auth 0 2024-04-19T01:44:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 10 monclient: _check_auth_tickets 2024-04-19T01:44:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 1 -- 172.21.15.62:0/1216443517 <== mon.1 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1244427781 0 0) 0x7fd7f8001a00 con 0x7fd8080a5f00 2024-04-19T01:44:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:28.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-19T01:44:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 10 monclient: dump: 2024-04-19T01:44:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:28.969 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] but i'm connected to v1:172.21.15.71:6789/0, reconnecting 2024-04-19T01:44:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 1 -- 172.21.15.62:0/1216443517 >> v1:172.21.15.71:6789/0 conn(0x7fd8080a5f00 legacy=0x7fd80813bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-19T01:44:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 1 --2- 172.21.15.62:0/1216443517 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7f0003b30 0x7fd7f0003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:28.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 10 monclient(hunting): picked mon.a con 0x7fd7f0003b30 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:44:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 1 --2- 172.21.15.62:0/1216443517 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7f0004810 0x7fd7f0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 10 monclient(hunting): picked mon.c con 0x7fd7f0004810 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:44:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 1 --2- 172.21.15.62:0/1216443517 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd7f00053b0 0x7fd7f000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 10 monclient(hunting): picked mon.b con 0x7fd7f00053b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:44:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 1 -- 172.21.15.62:0/1216443517 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd80806f9f0 con 0x7fd7f0003b30 2024-04-19T01:44:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 1 -- 172.21.15.62:0/1216443517 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd80806e020 con 0x7fd7f00053b0 2024-04-19T01:44:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 1 -- 172.21.15.62:0/1216443517 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fd7f000c370 con 0x7fd7f0004810 2024-04-19T01:44:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8067fc640 10 monclient(hunting): _finish_auth 0 2024-04-19T01:44:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd806ffd640 1 --2- 172.21.15.62:0/1216443517 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7f0004810 0x7fd7f0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8077fe640 1 --2- 172.21.15.62:0/1216443517 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7f0003b30 0x7fd7f0003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:28.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd806ffd640 10 monclient(hunting): get_auth_request con 0x7fd7f0004810 auth_method 0 2024-04-19T01:44:28.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd806ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:28.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd806ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:28.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd806ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:28.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8077fe640 10 monclient(hunting): get_auth_request con 0x7fd7f0003b30 auth_method 0 2024-04-19T01:44:28.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:28.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8077fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:28.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8077fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:28.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd806ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:28.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd806ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:28.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd806ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T01:44:28.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:28.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:28.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd807fff640 1 --2- 172.21.15.62:0/1216443517 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd7f00053b0 0x7fd7f000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:28.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd8077fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T01:44:28.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd807fff640 10 monclient(hunting): get_auth_request con 0x7fd7f00053b0 auth_method 0 2024-04-19T01:44:28.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd807fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd807fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd807fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd806ffd640 10 monclient(hunting): handle_auth_done global_id 4660 payload 293 2024-04-19T01:44:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd806ffd640 1 -- 172.21.15.62:0/1216443517 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd7f00053b0 msgr2=0x7fd7f000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:44:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd806ffd640 1 --2- 172.21.15.62:0/1216443517 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd7f00053b0 0x7fd7f000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd806ffd640 1 -- 172.21.15.62:0/1216443517 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7f0003b30 msgr2=0x7fd7f0003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:44:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd806ffd640 1 --2- 172.21.15.62:0/1216443517 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7f0003b30 0x7fd7f0003f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd806ffd640 10 monclient: _finish_hunting 0 2024-04-19T01:44:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd806ffd640 1 monclient: found mon.c 2024-04-19T01:44:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd806ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd806ffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:44:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd806ffd640 1 -- 172.21.15.62:0/1216443517 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7f0003530 con 0x7fd7f0004810 2024-04-19T01:44:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd806ffd640 10 monclient: _finish_auth 0 2024-04-19T01:44:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd806ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:44:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd806ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd80e21b640 5 monclient: authenticate success, global_id 4660 2024-04-19T01:44:28.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd80e21b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:44:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.969+0000 7fd807fff640 1 -- 172.21.15.62:0/1216443517 reap_dead start 2024-04-19T01:44:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.977+0000 7fd806ffd640 1 -- 172.21.15.62:0/1216443517 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7f0004810 msgr2=0x7fd7f0004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:28.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.977+0000 7fd806ffd640 1 --2- 172.21.15.62:0/1216443517 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7f0004810 0x7fd7f0004c90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8001c20 tx=0x7fd7f801cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcfe477540159e1c server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:28.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.977+0000 7fd8067fc640 1 -- 172.21.15.62:0/1216443517 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f800c0e0 con 0x7fd7f0004810 2024-04-19T01:44:28.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.977+0000 7fd8067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:28.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.977+0000 7fd8067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:44:28.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.977+0000 7fd8067fc640 10 monclient: dump: 2024-04-19T01:44:28.975 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:28.975 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:28.975 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:28.975 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:28.975 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:28.975 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:28.975 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:28.975 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:28.975 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:28.975 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:28.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.977+0000 7fd80e21b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:44:28.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.977+0000 7fd8067fc640 1 -- 172.21.15.62:0/1216443517 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7f80a9070 con 0x7fd7f0004810 2024-04-19T01:44:28.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.977+0000 7fd8067fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:28.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.977+0000 7fd80e21b640 10 monclient: get_monmap_and_config success 2024-04-19T01:44:28.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.977+0000 7fd80e21b640 4 set_mon_vals no callback set 2024-04-19T01:44:28.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.977+0000 7fd8067fc640 1 -- 172.21.15.62:0/1216443517 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f80a5cf0 con 0x7fd7f0004810 2024-04-19T01:44:28.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.977+0000 7fd8067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:28.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.977+0000 7fd8067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:44:28.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.977+0000 7fd8067fc640 10 monclient: dump: 2024-04-19T01:44:28.976 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:28.976 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:28.976 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:28.976 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:28.976 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:28.976 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:28.976 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:28.976 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:28.976 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:28.977 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.977+0000 7fd80e21b640 10 monclient: shutdown 2024-04-19T01:44:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.977+0000 7fd80e21b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.977+0000 7fd80e21b640 1 -- 172.21.15.62:0/1216443517 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7f0004810 msgr2=0x7fd7f0004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.977+0000 7fd80e21b640 1 --2- 172.21.15.62:0/1216443517 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7f0004810 0x7fd7f0004c90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8001c20 tx=0x7fd7f801cbf0 comp rx=0 tx=0).stop 2024-04-19T01:44:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.977+0000 7fd80e21b640 1 -- 172.21.15.62:0/1216443517 shutdown_connections 2024-04-19T01:44:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.977+0000 7fd80e21b640 1 --2- 172.21.15.62:0/1216443517 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd7f0004810 0x7fd7f0004c90 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.977+0000 7fd80e21b640 1 -- 172.21.15.62:0/1216443517 >> 172.21.15.62:0/1216443517 conn(0x7fd8080aa1d0 msgr2=0x7fd8080a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.977+0000 7fd80e21b640 1 -- 172.21.15.62:0/1216443517 shutdown_connections 2024-04-19T01:44:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.977+0000 7fd80e21b640 1 -- 172.21.15.62:0/1216443517 wait complete. 2024-04-19T01:44:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd80e21b640 10 monclient: build_initial_monmap 2024-04-19T01:44:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd80e21b640 10 monclient: monmap: 2024-04-19T01:44:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:28.980 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:44:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:44:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:44:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:44:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:44:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:44:28.981 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd80e21b640 1 Processor -- start 2024-04-19T01:44:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd80e21b640 1 -- start start 2024-04-19T01:44:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd80e21b640 10 monclient: init 2024-04-19T01:44:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd80e21b640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd80e21b640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-19T01:44:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd80e21b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7f0004810 0x7fd8081d4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd80e21b640 10 monclient(hunting): picked mon.noname-a con 0x7fd7f0004810 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:44:28.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd80e21b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8081d4bc0 0x7fd80809bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd80e21b640 10 monclient(hunting): picked mon.noname-b con 0x7fd8081d4bc0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:44:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd80e21b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd8081d5000 0x7fd80809c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd80e21b640 10 monclient(hunting): picked mon.noname-c con 0x7fd8081d5000 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:44:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd80e21b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd80e21b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd8080a71c0 con 0x7fd7f0004810 2024-04-19T01:44:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd80e21b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd80e21b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd8080a7040 con 0x7fd8081d4bc0 2024-04-19T01:44:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd80e21b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd80e21b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fd808070990 con 0x7fd8081d5000 2024-04-19T01:44:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd80e21b640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd80e21b640 10 monclient(hunting): authenticate will time out at 1054.801695s 2024-04-19T01:44:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd807fff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd8081d5000 0x7fd80809c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd8077fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7f0004810 0x7fd8081d4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:28.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd807fff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd8081d5000 0x7fd80809c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:40754/0 (socket says 172.21.15.62:40754) 2024-04-19T01:44:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd807fff640 1 -- 172.21.15.62:0/179602086 learned_addr learned my addr 172.21.15.62:0/179602086 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:44:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd8077fe640 10 monclient(hunting): get_auth_request con 0x7fd7f0004810 auth_method 0 2024-04-19T01:44:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd8077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd8077fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd8077fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd807fff640 10 monclient(hunting): get_auth_request con 0x7fd8081d5000 auth_method 0 2024-04-19T01:44:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd807fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd807fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd807fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd8077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd8077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd8077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd806ffd640 1 --2- 172.21.15.62:0/179602086 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8081d4bc0 0x7fd80809bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd807fff640 1 -- 172.21.15.62:0/179602086 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd8081d5000 msgr2=0x7fd80809c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-19T01:44:28.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd807fff640 1 -- 172.21.15.62:0/179602086 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd8081d5000 msgr2=0x7fd80809c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:44:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd807fff640 1 --2- 172.21.15.62:0/179602086 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd8081d5000 0x7fd80809c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd806ffd640 10 monclient(hunting): get_auth_request con 0x7fd8081d4bc0 auth_method 0 2024-04-19T01:44:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd806ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd806ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd806ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd807fff640 1 --2- 172.21.15.62:0/179602086 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd8081d5000 0x7fd80809c530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:44:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd8077fe640 10 monclient(hunting): handle_auth_done global_id 14103 payload 824 2024-04-19T01:44:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd8077fe640 1 -- 172.21.15.62:0/179602086 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd8081d5000 msgr2=0x7fd80809c530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:44:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd8077fe640 1 --2- 172.21.15.62:0/179602086 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd8081d5000 0x7fd80809c530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd8077fe640 1 -- 172.21.15.62:0/179602086 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8081d4bc0 msgr2=0x7fd80809bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd8077fe640 1 --2- 172.21.15.62:0/179602086 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8081d4bc0 0x7fd80809bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd8077fe640 10 monclient: _finish_hunting 0 2024-04-19T01:44:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd8077fe640 1 monclient: found mon.noname-a 2024-04-19T01:44:28.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd8077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:28.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd8077fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:44:28.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd8077fe640 1 -- 172.21.15.62:0/179602086 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd80809fca0 con 0x7fd7f0004810 2024-04-19T01:44:28.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.981+0000 7fd806ffd640 1 --2- 172.21.15.62:0/179602086 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8081d4bc0 0x7fd80809bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T01:44:28.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.985+0000 7fd8077fe640 1 -- 172.21.15.62:0/179602086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7f0004810 msgr2=0x7fd8081d4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:28.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.985+0000 7fd8077fe640 1 --2- 172.21.15.62:0/179602086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7f0004810 0x7fd8081d4680 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc000ee0 tx=0x7fd7fc017f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:28.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.985+0000 7fd7effff640 1 -- 172.21.15.62:0/179602086 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc00f040 con 0x7fd7f0004810 2024-04-19T01:44:28.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.985+0000 7fd7effff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:28.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.985+0000 7fd7effff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:44:28.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.985+0000 7fd7effff640 10 monclient: dump: 2024-04-19T01:44:28.987 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:28.987 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:28.988 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:28.988 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:28.988 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:28.988 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:28.988 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:28.988 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:28.988 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:28.988 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:28.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.985+0000 7fd7effff640 10 monclient: _finish_auth 0 2024-04-19T01:44:28.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.985+0000 7fd7effff640 10 monclient: _check_auth_tickets 2024-04-19T01:44:28.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.985+0000 7fd7effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:28.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.985+0000 7fd7effff640 1 -- 172.21.15.62:0/179602086 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7fc01c070 con 0x7fd7f0004810 2024-04-19T01:44:28.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.985+0000 7fd80e21b640 5 monclient: authenticate success, global_id 14103 2024-04-19T01:44:28.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.985+0000 7fd80e21b640 10 monclient: _renew_subs 2024-04-19T01:44:28.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.985+0000 7fd80e21b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:44:28.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.985+0000 7fd80e21b640 1 -- 172.21.15.62:0/179602086 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd80809ff80 con 0x7fd7f0004810 2024-04-19T01:44:28.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.985+0000 7fd7effff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:28.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.985+0000 7fd7effff640 1 -- 172.21.15.62:0/179602086 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc004270 con 0x7fd7f0004810 2024-04-19T01:44:28.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.985+0000 7fd7effff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:28.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.985+0000 7fd7effff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:44:28.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.985+0000 7fd7effff640 10 monclient: dump: 2024-04-19T01:44:28.989 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:28.989 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:28.989 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:28.989 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:28.989 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:28.989 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:28.989 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:28.989 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:28.989 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:28.989 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:28.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.985+0000 7fd805ffb640 4 set_mon_vals no callback set 2024-04-19T01:44:28.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.985+0000 7fd80e21b640 10 monclient: _renew_subs 2024-04-19T01:44:28.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.985+0000 7fd80e21b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:44:28.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.985+0000 7fd80e21b640 1 -- 172.21.15.62:0/179602086 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd80809dc90 con 0x7fd7f0004810 2024-04-19T01:44:28.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.985+0000 7fd80e21b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:28.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.985+0000 7fd80e21b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:28.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.985+0000 7fd80e21b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:44:28.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.985+0000 7fd80e21b640 1 -- 172.21.15.62:0/179602086 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd808068ab0 con 0x7fd7f0004810 2024-04-19T01:44:28.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.985+0000 7fd7effff640 1 -- 172.21.15.62:0/179602086 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fd7fc003620 con 0x7fd7f0004810 2024-04-19T01:44:28.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.985+0000 7fd7effff640 1 --2- 172.21.15.62:0/179602086 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fd7f4036420 0x7fd7f40388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:28.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.989+0000 7fd7effff640 1 -- 172.21.15.62:0/179602086 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 4738+0+0 (secure 0 0 0) 0x7fd7fc04a4d0 con 0x7fd7f0004810 2024-04-19T01:44:28.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.989+0000 7fd806ffd640 1 --2- 172.21.15.62:0/179602086 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fd7f4036420 0x7fd7f40388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:28.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.989+0000 7fd806ffd640 10 monclient: get_auth_request con 0x7fd7f4036420 auth_method 0 2024-04-19T01:44:28.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.989+0000 7fd806ffd640 1 --2- 172.21.15.62:0/179602086 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fd7f4036420 0x7fd7f40388e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8001ec0 tx=0x7fd7f80b1040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:28.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.993+0000 7fd7effff640 1 -- 172.21.15.62:0/179602086 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd7fc02a7f0 con 0x7fd7f0004810 2024-04-19T01:44:28.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.993+0000 7fd7effff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:28.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:28.993+0000 7fd7effff640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:44:29.039 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28785]: starting. 2024-04-19T01:44:29.039 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28785]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-28783 2024-04-19T01:44:29.039 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28785]: shutting down. 2024-04-19T01:44:29.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:29.221+0000 7fd80e21b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:44:29.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:29.221+0000 7fd80e21b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:44:29.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:29.221+0000 7fd80e21b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:44:29.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:29.221+0000 7fd80e21b640 1 -- 172.21.15.62:0/179602086 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd808068710 con 0x7fd7f0004810 2024-04-19T01:44:29.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:29.221+0000 7fd7effff640 1 -- 172.21.15.62:0/179602086 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7fd7fc047f40 con 0x7fd7f0004810 2024-04-19T01:44:29.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:29.221+0000 7fd7effff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:44:29.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:29.221+0000 7fd7effff640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:44:29.218 INFO:teuthology.orchestra.run.smithi062.stdout:{"election_epoch":18,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","modified":"2024-04-19T01:41:38.497795Z","created":"2024-04-19T01:41:38.497795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:44:29.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:29.221+0000 7fd80e21b640 1 -- 172.21.15.62:0/179602086 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fd7f4036420 msgr2=0x7fd7f40388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:29.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:29.221+0000 7fd80e21b640 1 --2- 172.21.15.62:0/179602086 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fd7f4036420 0x7fd7f40388e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8001ec0 tx=0x7fd7f80b1040 comp rx=0 tx=0).stop 2024-04-19T01:44:29.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:29.221+0000 7fd80e21b640 10 monclient: shutdown 2024-04-19T01:44:29.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:29.221+0000 7fd80e21b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:29.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:29.221+0000 7fd80e21b640 1 -- 172.21.15.62:0/179602086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7f0004810 msgr2=0x7fd8081d4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:29.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:29.221+0000 7fd80e21b640 1 --2- 172.21.15.62:0/179602086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7f0004810 0x7fd8081d4680 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc000ee0 tx=0x7fd7fc017f50 comp rx=0 tx=0).stop 2024-04-19T01:44:29.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:29.221+0000 7fd80e21b640 1 -- 172.21.15.62:0/179602086 shutdown_connections 2024-04-19T01:44:29.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:29.221+0000 7fd80e21b640 1 --2- 172.21.15.62:0/179602086 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fd7f4036420 0x7fd7f40388e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:29.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:29.221+0000 7fd80e21b640 1 --2- 172.21.15.62:0/179602086 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd8081d5000 0x7fd80809c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:29.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:29.221+0000 7fd80e21b640 1 --2- 172.21.15.62:0/179602086 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8081d4bc0 0x7fd80809bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:29.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:29.221+0000 7fd80e21b640 1 --2- 172.21.15.62:0/179602086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7f0004810 0x7fd8081d4680 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:29.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:29.221+0000 7fd80e21b640 1 -- 172.21.15.62:0/179602086 >> 172.21.15.62:0/179602086 conn(0x7fd8080aa1d0 msgr2=0x7fd80813b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:29.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:29.221+0000 7fd80e21b640 1 -- 172.21.15.62:0/179602086 shutdown_connections 2024-04-19T01:44:29.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:29.221+0000 7fd80e21b640 1 -- 172.21.15.62:0/179602086 wait complete. 2024-04-19T01:44:32.235 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:44:32.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.373+0000 7fd411f03640 10 monclient: get_monmap_and_config 2024-04-19T01:44:32.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.373+0000 7fd411f03640 10 monclient: build_initial_monmap 2024-04-19T01:44:32.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.373+0000 7fd411f03640 10 monclient: monmap: 2024-04-19T01:44:32.371 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:32.371 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:32.371 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:44:32.376107+0000 2024-04-19T01:44:32.371 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:44:32.376107+0000 2024-04-19T01:44:32.371 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:32.371 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:32.371 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:44:32.371 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:44:32.371 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:44:32.371 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:44:32.372 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:44:32.372 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:32.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.373+0000 7fd411f03640 1 Processor -- start 2024-04-19T01:44:32.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.373+0000 7fd411f03640 1 -- start start 2024-04-19T01:44:32.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.373+0000 7fd411f03640 10 monclient: init 2024-04-19T01:44:32.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.373+0000 7fd411f03640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:32.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.373+0000 7fd411f03640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-04-19T01:44:32.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.377+0000 7fd411f03640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd40c0a47e0 addr v1:172.21.15.71:6789/0 2024-04-19T01:44:32.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.377+0000 7fd411f03640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd40c0a4bc0 addr v1:172.21.15.62:6789/0 2024-04-19T01:44:32.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.377+0000 7fd411f03640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd40c13a570 0x7fd40c13c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:32.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.377+0000 7fd411f03640 10 monclient(hunting): picked mon.noname-c con 0x7fd40c13a570 addr v2:172.21.15.71:3300/0 2024-04-19T01:44:32.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.377+0000 7fd411f03640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd40c09e6e0 con 0x7fd40c0a4bc0 2024-04-19T01:44:32.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.377+0000 7fd411f03640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd40c09e9c0 con 0x7fd40c0a47e0 2024-04-19T01:44:32.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.377+0000 7fd411f03640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:32.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.377+0000 7fd411f03640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fd40c09d4e0 con 0x7fd40c13a570 2024-04-19T01:44:32.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.377+0000 7fd411f03640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:32.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.377+0000 7fd411f03640 10 monclient(hunting): authenticate will time out at 1058.196117s 2024-04-19T01:44:32.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.377+0000 7fd40affd640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fd40c0a4bc0 0x7fd40c0a4ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35154/0 (socket says 172.21.15.62:35154) 2024-04-19T01:44:32.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.377+0000 7fd40affd640 1 -- 172.21.15.62:0/3119459846 learned_addr learned my addr 172.21.15.62:0/3119459846 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:44:32.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.377+0000 7fd40bfff640 1 --2- 172.21.15.62:0/3119459846 >> v2:172.21.15.71:3300/0 conn(0x7fd40c13a570 0x7fd40c13c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:32.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.377+0000 7fd40bfff640 10 monclient(hunting): get_auth_request con 0x7fd40c13a570 auth_method 0 2024-04-19T01:44:32.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.377+0000 7fd40bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:32.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.377+0000 7fd40bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:32.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.377+0000 7fd40bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:32.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.377+0000 7fd40bfff640 1 -- 172.21.15.62:0/3119459846 >> v2:172.21.15.71:3300/0 conn(0x7fd40c13a570 msgr2=0x7fd40c13c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:44:32.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.377+0000 7fd40bfff640 1 -- 172.21.15.62:0/3119459846 >> v2:172.21.15.71:3300/0 conn(0x7fd40c13a570 msgr2=0x7fd40c13c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:32.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.377+0000 7fd40bfff640 1 --2- 172.21.15.62:0/3119459846 >> v2:172.21.15.71:3300/0 conn(0x7fd40c13a570 0x7fd40c13c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:32.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.377+0000 7fd40bfff640 1 --2- 172.21.15.62:0/3119459846 >> v2:172.21.15.71:3300/0 conn(0x7fd40c13a570 0x7fd40c13c960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:44:32.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.377+0000 7fd40affd640 1 -- 172.21.15.62:0/3119459846 >> v1:172.21.15.62:6789/0 conn(0x7fd40c0a4bc0 legacy=0x7fd40c0a4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:32.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.377+0000 7fd40b7fe640 1 -- 172.21.15.62:0/3119459846 >> v1:172.21.15.71:6789/0 conn(0x7fd40c0a47e0 legacy=0x7fd40c09f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:32.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.581+0000 7fd40bfff640 1 --2- 172.21.15.62:0/3119459846 >> v2:172.21.15.71:3300/0 conn(0x7fd40c13a570 0x7fd40c13c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:32.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.581+0000 7fd40bfff640 10 monclient(hunting): get_auth_request con 0x7fd40c13a570 auth_method 0 2024-04-19T01:44:32.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.581+0000 7fd40bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:32.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.581+0000 7fd40bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:32.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.581+0000 7fd40bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:32.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.581+0000 7fd40bfff640 1 -- 172.21.15.62:0/3119459846 >> v2:172.21.15.71:3300/0 conn(0x7fd40c13a570 msgr2=0x7fd40c13c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:44:32.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.581+0000 7fd40bfff640 1 -- 172.21.15.62:0/3119459846 >> v2:172.21.15.71:3300/0 conn(0x7fd40c13a570 msgr2=0x7fd40c13c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:32.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.581+0000 7fd40bfff640 1 --2- 172.21.15.62:0/3119459846 >> v2:172.21.15.71:3300/0 conn(0x7fd40c13a570 0x7fd40c13c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:32.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.581+0000 7fd40bfff640 1 --2- 172.21.15.62:0/3119459846 >> v2:172.21.15.71:3300/0 conn(0x7fd40c13a570 0x7fd40c13c960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:44:32.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.985+0000 7fd40bfff640 1 --2- 172.21.15.62:0/3119459846 >> v2:172.21.15.71:3300/0 conn(0x7fd40c13a570 0x7fd40c13c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:32.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.985+0000 7fd40bfff640 10 monclient(hunting): get_auth_request con 0x7fd40c13a570 auth_method 0 2024-04-19T01:44:32.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.985+0000 7fd40bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:32.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.985+0000 7fd40bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:32.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.985+0000 7fd40bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:32.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.985+0000 7fd40bfff640 1 -- 172.21.15.62:0/3119459846 >> v2:172.21.15.71:3300/0 conn(0x7fd40c13a570 msgr2=0x7fd40c13c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:44:32.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.985+0000 7fd40bfff640 1 -- 172.21.15.62:0/3119459846 >> v2:172.21.15.71:3300/0 conn(0x7fd40c13a570 msgr2=0x7fd40c13c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:32.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.985+0000 7fd40bfff640 1 --2- 172.21.15.62:0/3119459846 >> v2:172.21.15.71:3300/0 conn(0x7fd40c13a570 0x7fd40c13c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:32.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:32.985+0000 7fd40bfff640 1 --2- 172.21.15.62:0/3119459846 >> v2:172.21.15.71:3300/0 conn(0x7fd40c13a570 0x7fd40c13c960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:44:33.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:33.789+0000 7fd40bfff640 1 --2- 172.21.15.62:0/3119459846 >> v2:172.21.15.71:3300/0 conn(0x7fd40c13a570 0x7fd40c13c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:33.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:33.789+0000 7fd40bfff640 10 monclient(hunting): get_auth_request con 0x7fd40c13a570 auth_method 0 2024-04-19T01:44:33.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:33.789+0000 7fd40bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:33.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:33.789+0000 7fd40bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:33.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:33.789+0000 7fd40bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:33.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:33.789+0000 7fd40bfff640 1 -- 172.21.15.62:0/3119459846 >> v2:172.21.15.71:3300/0 conn(0x7fd40c13a570 msgr2=0x7fd40c13c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:44:33.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:33.789+0000 7fd40bfff640 1 -- 172.21.15.62:0/3119459846 >> v2:172.21.15.71:3300/0 conn(0x7fd40c13a570 msgr2=0x7fd40c13c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:33.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:33.789+0000 7fd40bfff640 1 --2- 172.21.15.62:0/3119459846 >> v2:172.21.15.71:3300/0 conn(0x7fd40c13a570 0x7fd40c13c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:33.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:33.789+0000 7fd40bfff640 1 --2- 172.21.15.62:0/3119459846 >> v2:172.21.15.71:3300/0 conn(0x7fd40c13a570 0x7fd40c13c960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:44:35.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 1 -- 172.21.15.62:0/3119459846 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3151884916 0 0) 0x7fd40c09e9c0 con 0x7fd40c0a47e0 2024-04-19T01:44:35.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 10 monclient(hunting): _negotiate 2024-04-19T01:44:35.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:35.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:35.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 10 monclient(hunting): my global_id is 4684 2024-04-19T01:44:35.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 1 -- 172.21.15.62:0/3119459846 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3f4003530 con 0x7fd40c0a47e0 2024-04-19T01:44:35.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 1 -- 172.21.15.62:0/3119459846 <== mon.1 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1645421007 0 0) 0x7fd3f4003530 con 0x7fd40c0a47e0 2024-04-19T01:44:35.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 1 -- 172.21.15.62:0/3119459846 >> v2:172.21.15.71:3300/0 conn(0x7fd40c13a570 msgr2=0x7fd40c13c960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:44:35.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 1 --2- 172.21.15.62:0/3119459846 >> v2:172.21.15.71:3300/0 conn(0x7fd40c13a570 0x7fd40c13c960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:35.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 1 -- 172.21.15.62:0/3119459846 >> v1:172.21.15.62:6789/0 conn(0x7fd40c0a4bc0 legacy=0x7fd40c0a4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:35.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 10 monclient: _finish_hunting 0 2024-04-19T01:44:35.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 1 monclient: found mon.noname-c-legacy 2024-04-19T01:44:35.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:35.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.71:6789/0 2024-04-19T01:44:35.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 1 -- 172.21.15.62:0/3119459846 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd40c0a3150 con 0x7fd40c0a47e0 2024-04-19T01:44:35.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 10 monclient: _finish_auth 0 2024-04-19T01:44:35.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 10 monclient: _check_auth_tickets 2024-04-19T01:44:35.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:35.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 1 -- 172.21.15.62:0/3119459846 <== mon.1 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1244427781 0 0) 0x7fd3fc001a00 con 0x7fd40c0a47e0 2024-04-19T01:44:35.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:35.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-19T01:44:35.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 10 monclient: dump: 2024-04-19T01:44:35.007 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:35.007 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:35.007 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:35.007 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:35.007 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:35.008 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:35.008 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:35.008 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:35.008 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:35.008 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:35.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] but i'm connected to v1:172.21.15.71:6789/0, reconnecting 2024-04-19T01:44:35.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:35.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 1 -- 172.21.15.62:0/3119459846 >> v1:172.21.15.71:6789/0 conn(0x7fd40c0a47e0 legacy=0x7fd40c09f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:35.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-19T01:44:35.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 1 --2- 172.21.15.62:0/3119459846 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3f4003b30 0x7fd3f4003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:35.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 10 monclient(hunting): picked mon.b con 0x7fd3f4003b30 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:44:35.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 1 --2- 172.21.15.62:0/3119459846 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3f4004810 0x7fd3f4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:35.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 10 monclient(hunting): picked mon.c con 0x7fd3f4004810 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:44:35.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 1 --2- 172.21.15.62:0/3119459846 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3f4005420 0x7fd3f400ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:35.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 10 monclient(hunting): picked mon.a con 0x7fd3f4005420 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:44:35.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:35.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 1 -- 172.21.15.62:0/3119459846 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd3f400c300 con 0x7fd3f4005420 2024-04-19T01:44:35.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:35.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 1 -- 172.21.15.62:0/3119459846 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd3f400c4b0 con 0x7fd3f4003b30 2024-04-19T01:44:35.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:35.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 1 -- 172.21.15.62:0/3119459846 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fd3f400c6b0 con 0x7fd3f4004810 2024-04-19T01:44:35.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:35.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.005+0000 7fd40a7fc640 10 monclient(hunting): _finish_auth 0 2024-04-19T01:44:35.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.009+0000 7fd40affd640 1 --2- 172.21.15.62:0/3119459846 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3f4004810 0x7fd3f4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:35.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.009+0000 7fd40b7fe640 1 --2- 172.21.15.62:0/3119459846 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3f4003b30 0x7fd3f4003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:35.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.009+0000 7fd40affd640 10 monclient(hunting): get_auth_request con 0x7fd3f4004810 auth_method 0 2024-04-19T01:44:35.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.009+0000 7fd40affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:35.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.009+0000 7fd40affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:35.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.009+0000 7fd40affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:35.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.009+0000 7fd40b7fe640 10 monclient(hunting): get_auth_request con 0x7fd3f4003b30 auth_method 0 2024-04-19T01:44:35.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.009+0000 7fd40b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:35.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.009+0000 7fd40b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:35.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.009+0000 7fd40b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:35.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.009+0000 7fd40b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:35.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.009+0000 7fd40b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:35.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.009+0000 7fd40b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T01:44:35.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.009+0000 7fd40affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:35.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.009+0000 7fd40affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:35.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.009+0000 7fd40affd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T01:44:35.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.009+0000 7fd40b7fe640 10 monclient(hunting): handle_auth_done global_id 4684 payload 293 2024-04-19T01:44:35.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.009+0000 7fd40b7fe640 1 -- 172.21.15.62:0/3119459846 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3f4004810 msgr2=0x7fd3f4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:44:35.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.009+0000 7fd40b7fe640 1 --2- 172.21.15.62:0/3119459846 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3f4004810 0x7fd3f4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:35.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.009+0000 7fd40b7fe640 1 -- 172.21.15.62:0/3119459846 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3f4005420 msgr2=0x7fd3f400ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:44:35.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.009+0000 7fd40b7fe640 1 --2- 172.21.15.62:0/3119459846 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3f4005420 0x7fd3f400ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:35.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.009+0000 7fd40bfff640 1 -- 172.21.15.62:0/3119459846 reap_dead start 2024-04-19T01:44:35.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.009+0000 7fd40b7fe640 10 monclient: _finish_hunting 0 2024-04-19T01:44:35.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.009+0000 7fd40b7fe640 1 monclient: found mon.b 2024-04-19T01:44:35.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.009+0000 7fd40b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:35.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.009+0000 7fd40b7fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:44:35.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.009+0000 7fd40b7fe640 1 -- 172.21.15.62:0/3119459846 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3f4003530 con 0x7fd3f4003b30 2024-04-19T01:44:35.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.009+0000 7fd40b7fe640 10 monclient: _finish_auth 0 2024-04-19T01:44:35.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.009+0000 7fd40b7fe640 10 monclient: _check_auth_tickets 2024-04-19T01:44:35.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.009+0000 7fd40b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:35.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.009+0000 7fd411f03640 5 monclient: authenticate success, global_id 4684 2024-04-19T01:44:35.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.009+0000 7fd411f03640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:44:35.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.013+0000 7fd40b7fe640 1 -- 172.21.15.62:0/3119459846 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3f4003b30 msgr2=0x7fd3f4003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:35.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.013+0000 7fd40b7fe640 1 --2- 172.21.15.62:0/3119459846 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3f4003b30 0x7fd3f4003f50 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc001c20 tx=0x7fd3fc01f390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c09bcbef66679624 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:35.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.013+0000 7fd40a7fc640 1 -- 172.21.15.62:0/3119459846 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc00c0e0 con 0x7fd3f4003b30 2024-04-19T01:44:35.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.013+0000 7fd40a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:35.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.013+0000 7fd40a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:44:35.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.013+0000 7fd40a7fc640 10 monclient: dump: 2024-04-19T01:44:35.014 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:35.014 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:35.014 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:35.014 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:35.014 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:35.015 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:35.015 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:35.015 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:35.015 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:35.015 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:35.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.013+0000 7fd411f03640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:44:35.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.013+0000 7fd40b7fe640 1 --2- 172.21.15.62:0/3119459846 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3f4003b30 0x7fd3f4003f50 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc001c20 tx=0x7fd3fc01f390 comp rx=0 tx=0).queue_received will delay after 1.71349e+09 on 0x7fd3fc007270 config(0 keys) 2024-04-19T01:44:35.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.013+0000 7fd40a7fc640 1 -- 172.21.15.62:0/3119459846 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3fc007270 con 0x7fd3f4003b30 2024-04-19T01:44:35.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.013+0000 7fd40a7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:35.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.013+0000 7fd411f03640 10 monclient: get_monmap_and_config success 2024-04-19T01:44:35.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.013+0000 7fd411f03640 4 set_mon_vals no callback set 2024-04-19T01:44:35.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd411f03640 10 monclient: shutdown 2024-04-19T01:44:35.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd411f03640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:35.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd411f03640 1 -- 172.21.15.62:0/3119459846 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3f4003b30 msgr2=0x7fd3f4003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:35.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd411f03640 1 --2- 172.21.15.62:0/3119459846 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3f4003b30 0x7fd3f4003f50 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc001c20 tx=0x7fd3fc01f390 comp rx=0 tx=0).stop 2024-04-19T01:44:35.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd40a7fc640 1 -- 172.21.15.62:0/3119459846 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 0 0 0) 0x7fd3fc007660 con 0x7fd3f4003b30 2024-04-19T01:44:35.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd40a7fc640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-04-19T01:44:35.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd411f03640 1 -- 172.21.15.62:0/3119459846 shutdown_connections 2024-04-19T01:44:35.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd411f03640 1 --2- 172.21.15.62:0/3119459846 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd3f4003b30 0x7fd3f4003f50 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:35.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd411f03640 1 -- 172.21.15.62:0/3119459846 >> 172.21.15.62:0/3119459846 conn(0x7fd40c0a23d0 msgr2=0x7fd40c0a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:35.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd411f03640 1 -- 172.21.15.62:0/3119459846 shutdown_connections 2024-04-19T01:44:35.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd411f03640 1 -- 172.21.15.62:0/3119459846 wait complete. 2024-04-19T01:44:35.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd411f03640 10 monclient: build_initial_monmap 2024-04-19T01:44:35.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd411f03640 10 monclient: monmap: 2024-04-19T01:44:35.020 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:35.020 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:44:35.020 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:44:35.020 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:44:35.020 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:35.020 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:35.020 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:44:35.020 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:44:35.020 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:44:35.021 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:35.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd411f03640 1 Processor -- start 2024-04-19T01:44:35.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd411f03640 1 -- start start 2024-04-19T01:44:35.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd411f03640 10 monclient: init 2024-04-19T01:44:35.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd411f03640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:35.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd411f03640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-19T01:44:35.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd411f03640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3f4003b30 0x7fd40c1cc990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:35.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd411f03640 10 monclient(hunting): picked mon.noname-c con 0x7fd3f4003b30 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:44:35.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd411f03640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd40c138110 0x7fd40c1cced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:35.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd411f03640 10 monclient(hunting): picked mon.noname-a con 0x7fd40c138110 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:44:35.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd411f03640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd40c1226d0 0x7fd40c122b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:35.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd411f03640 10 monclient(hunting): picked mon.noname-b con 0x7fd40c1226d0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:44:35.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd411f03640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:35.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd411f03640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd40c09f7e0 con 0x7fd40c138110 2024-04-19T01:44:35.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd411f03640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:35.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd411f03640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd40c09f660 con 0x7fd40c1226d0 2024-04-19T01:44:35.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd411f03640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:35.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd411f03640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fd40c09f960 con 0x7fd3f4003b30 2024-04-19T01:44:35.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd411f03640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:35.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd411f03640 10 monclient(hunting): authenticate will time out at 1060.839166s 2024-04-19T01:44:35.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd40b7fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3f4003b30 0x7fd40c1cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:35.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd40affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd40c138110 0x7fd40c1cced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:35.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd40b7fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3f4003b30 0x7fd40c1cc990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:40772/0 (socket says 172.21.15.62:40772) 2024-04-19T01:44:35.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd40b7fe640 1 -- 172.21.15.62:0/338187360 learned_addr learned my addr 172.21.15.62:0/338187360 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:44:35.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd40b7fe640 10 monclient(hunting): get_auth_request con 0x7fd3f4003b30 auth_method 0 2024-04-19T01:44:35.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd40b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:35.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd40b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:35.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd40b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:35.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd40affd640 10 monclient(hunting): get_auth_request con 0x7fd40c138110 auth_method 0 2024-04-19T01:44:35.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd40affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:35.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd40affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:35.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd40affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:35.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd40bfff640 1 --2- 172.21.15.62:0/338187360 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd40c1226d0 0x7fd40c122b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:35.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd40b7fe640 1 -- 172.21.15.62:0/338187360 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3f4003b30 msgr2=0x7fd40c1cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-19T01:44:35.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd40bfff640 10 monclient(hunting): get_auth_request con 0x7fd40c1226d0 auth_method 0 2024-04-19T01:44:35.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd40b7fe640 1 -- 172.21.15.62:0/338187360 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3f4003b30 msgr2=0x7fd40c1cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:44:35.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd40bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:35.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd40bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:35.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd40bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:35.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd40b7fe640 1 --2- 172.21.15.62:0/338187360 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3f4003b30 0x7fd40c1cc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:35.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd40affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:35.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd40affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:35.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd40affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:35.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd40b7fe640 1 --2- 172.21.15.62:0/338187360 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3f4003b30 0x7fd40c1cc990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:44:35.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd40bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:35.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd40bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:35.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd40bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:35.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.017+0000 7fd40affd640 10 monclient(hunting): handle_auth_done global_id 14121 payload 824 2024-04-19T01:44:35.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd40affd640 1 -- 172.21.15.62:0/338187360 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3f4003b30 msgr2=0x7fd40c1cc990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:44:35.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd40affd640 1 --2- 172.21.15.62:0/338187360 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3f4003b30 0x7fd40c1cc990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:35.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd40affd640 1 -- 172.21.15.62:0/338187360 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd40c1226d0 msgr2=0x7fd40c122b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:35.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd40affd640 1 --2- 172.21.15.62:0/338187360 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd40c1226d0 0x7fd40c122b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:35.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd40affd640 10 monclient: _finish_hunting 0 2024-04-19T01:44:35.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd40affd640 1 monclient: found mon.noname-a 2024-04-19T01:44:35.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd40affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:35.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd40affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:44:35.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd40affd640 1 -- 172.21.15.62:0/338187360 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd40c126660 con 0x7fd40c138110 2024-04-19T01:44:35.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd40affd640 1 -- 172.21.15.62:0/338187360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd40c138110 msgr2=0x7fd40c1cced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:35.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd40affd640 1 --2- 172.21.15.62:0/338187360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd40c138110 0x7fd40c1cced0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd40c074f60 tx=0x7fd400007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:35.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd3f3fff640 1 -- 172.21.15.62:0/338187360 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4000117c0 con 0x7fd40c138110 2024-04-19T01:44:35.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd3f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:35.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd3f3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:44:35.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd3f3fff640 10 monclient: dump: 2024-04-19T01:44:35.025 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:35.025 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:35.025 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:35.025 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:35.025 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:35.026 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:35.026 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:35.026 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:35.026 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:35.026 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:35.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd3f3fff640 10 monclient: _finish_auth 0 2024-04-19T01:44:35.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd3f3fff640 10 monclient: _check_auth_tickets 2024-04-19T01:44:35.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd3f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:35.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd411f03640 5 monclient: authenticate success, global_id 14121 2024-04-19T01:44:35.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd411f03640 10 monclient: _renew_subs 2024-04-19T01:44:35.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd3f3fff640 1 -- 172.21.15.62:0/338187360 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd400011960 con 0x7fd40c138110 2024-04-19T01:44:35.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd411f03640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:44:35.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd411f03640 1 -- 172.21.15.62:0/338187360 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd40c124370 con 0x7fd40c138110 2024-04-19T01:44:35.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd3f3fff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:35.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd3f3fff640 1 -- 172.21.15.62:0/338187360 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd400011b50 con 0x7fd40c138110 2024-04-19T01:44:35.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd3f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:35.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd3f3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:44:35.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd3f3fff640 10 monclient: dump: 2024-04-19T01:44:35.027 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:35.027 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:35.027 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:35.027 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:35.027 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:35.027 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:35.027 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:35.027 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:35.027 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:35.027 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:35.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd411f03640 10 monclient: _renew_subs 2024-04-19T01:44:35.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd411f03640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:44:35.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd411f03640 1 -- 172.21.15.62:0/338187360 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd40c126a80 con 0x7fd40c138110 2024-04-19T01:44:35.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd409ffb640 4 set_mon_vals no callback set 2024-04-19T01:44:35.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.021+0000 7fd3f3fff640 1 -- 172.21.15.62:0/338187360 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fd40001a020 con 0x7fd40c138110 2024-04-19T01:44:35.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.025+0000 7fd411f03640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:35.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.025+0000 7fd411f03640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:35.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.025+0000 7fd411f03640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:44:35.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.025+0000 7fd411f03640 1 -- 172.21.15.62:0/338187360 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd40c0008e0 con 0x7fd40c138110 2024-04-19T01:44:35.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.025+0000 7fd3f3fff640 1 --2- 172.21.15.62:0/338187360 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fd3f80364a0 0x7fd3f8038960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:35.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.025+0000 7fd3f3fff640 1 -- 172.21.15.62:0/338187360 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 4346+0+0 (secure 0 0 0) 0x7fd4000488b0 con 0x7fd40c138110 2024-04-19T01:44:35.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.025+0000 7fd40b7fe640 1 --2- 172.21.15.62:0/338187360 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fd3f80364a0 0x7fd3f8038960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:35.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.025+0000 7fd40b7fe640 10 monclient: get_auth_request con 0x7fd3f80364a0 auth_method 0 2024-04-19T01:44:35.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.025+0000 7fd40b7fe640 1 --2- 172.21.15.62:0/338187360 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fd3f80364a0 0x7fd3f8038960 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc01da90 tx=0x7fd3fc007040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:35.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.029+0000 7fd3f3fff640 1 -- 172.21.15.62:0/338187360 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd400021020 con 0x7fd40c138110 2024-04-19T01:44:35.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.029+0000 7fd3f3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:35.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.029+0000 7fd3f3fff640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:44:35.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.265+0000 7fd411f03640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:44:35.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.265+0000 7fd411f03640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:44:35.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.265+0000 7fd411f03640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:44:35.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.265+0000 7fd411f03640 1 -- 172.21.15.62:0/338187360 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd40c125dc0 con 0x7fd40c138110 2024-04-19T01:44:35.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.269+0000 7fd3f3fff640 1 -- 172.21.15.62:0/338187360 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7fd40001a310 con 0x7fd40c138110 2024-04-19T01:44:35.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.269+0000 7fd3f3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:44:35.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.269+0000 7fd3f3fff640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:44:35.265 INFO:teuthology.orchestra.run.smithi062.stdout:{"election_epoch":22,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","modified":"2024-04-19T01:41:38.497795Z","created":"2024-04-19T01:41:38.497795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:44:35.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.269+0000 7fd411f03640 1 -- 172.21.15.62:0/338187360 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fd3f80364a0 msgr2=0x7fd3f8038960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:35.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.269+0000 7fd411f03640 1 --2- 172.21.15.62:0/338187360 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fd3f80364a0 0x7fd3f8038960 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc01da90 tx=0x7fd3fc007040 comp rx=0 tx=0).stop 2024-04-19T01:44:35.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.269+0000 7fd411f03640 10 monclient: shutdown 2024-04-19T01:44:35.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.269+0000 7fd411f03640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:35.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.269+0000 7fd411f03640 1 -- 172.21.15.62:0/338187360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd40c138110 msgr2=0x7fd40c1cced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:35.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.269+0000 7fd411f03640 1 --2- 172.21.15.62:0/338187360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd40c138110 0x7fd40c1cced0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd40c074f60 tx=0x7fd400007430 comp rx=0 tx=0).stop 2024-04-19T01:44:35.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.269+0000 7fd411f03640 1 -- 172.21.15.62:0/338187360 shutdown_connections 2024-04-19T01:44:35.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.269+0000 7fd411f03640 1 --2- 172.21.15.62:0/338187360 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd40c1226d0 0x7fd40c122b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:35.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.269+0000 7fd411f03640 1 --2- 172.21.15.62:0/338187360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd40c138110 0x7fd40c1cced0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:35.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.269+0000 7fd411f03640 1 --2- 172.21.15.62:0/338187360 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fd3f80364a0 0x7fd3f8038960 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:35.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.269+0000 7fd411f03640 1 --2- 172.21.15.62:0/338187360 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd3f4003b30 0x7fd40c1cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:35.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.269+0000 7fd411f03640 1 -- 172.21.15.62:0/338187360 >> 172.21.15.62:0/338187360 conn(0x7fd40c0a23d0 msgr2=0x7fd40c1399f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:35.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.269+0000 7fd411f03640 1 -- 172.21.15.62:0/338187360 shutdown_connections 2024-04-19T01:44:35.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:35.269+0000 7fd411f03640 1 -- 172.21.15.62:0/338187360 wait complete. 2024-04-19T01:44:38.285 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:44:38.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f5971edb640 10 monclient: get_monmap_and_config 2024-04-19T01:44:38.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f5971edb640 10 monclient: build_initial_monmap 2024-04-19T01:44:38.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f5971edb640 10 monclient: monmap: 2024-04-19T01:44:38.418 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:38.418 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:38.418 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:44:38.423129+0000 2024-04-19T01:44:38.418 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:44:38.423129+0000 2024-04-19T01:44:38.418 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:38.418 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:38.418 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:44:38.418 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:44:38.418 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:44:38.418 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:44:38.419 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:44:38.419 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:38.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f5971edb640 1 Processor -- start 2024-04-19T01:44:38.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f5971edb640 1 -- start start 2024-04-19T01:44:38.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f5971edb640 10 monclient: init 2024-04-19T01:44:38.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f5971edb640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:38.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f5971edb640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-04-19T01:44:38.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f5971edb640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f596c0a47e0 0x7f596c0a4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:38.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f5971edb640 10 monclient(hunting): picked mon.noname-c con 0x7f596c0a47e0 addr v2:172.21.15.71:3300/0 2024-04-19T01:44:38.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f5971edb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c09c990 0x7f596c0a5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:38.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f5971edb640 10 monclient(hunting): picked mon.noname-b con 0x7f596c09c990 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:44:38.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f5971edb640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f596c13b8a0 0x7f596c13bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:38.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f5971edb640 10 monclient(hunting): picked mon.noname-a con 0x7f596c13b8a0 addr v2:172.21.15.62:3300/0 2024-04-19T01:44:38.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f5971edb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:38.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f5971edb640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f596c09c760 con 0x7f596c13b8a0 2024-04-19T01:44:38.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f5971edb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:38.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f5971edb640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f596c09d8f0 con 0x7f596c0a47e0 2024-04-19T01:44:38.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f5971edb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:38.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f5971edb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f596c0a5640 con 0x7f596c09c990 2024-04-19T01:44:38.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f5971edb640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:38.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f5971edb640 10 monclient(hunting): authenticate will time out at 1064.242656s 2024-04-19T01:44:38.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f596bfff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f596c13b8a0 0x7f596c13bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:38.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f596affd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c09c990 0x7f596c0a5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:38.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f596bfff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f596c13b8a0 0x7f596c13bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44464/0 (socket says 172.21.15.62:44464) 2024-04-19T01:44:38.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f596affd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c09c990 0x7f596c0a5100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:51540/0 (socket says 172.21.15.62:51540) 2024-04-19T01:44:38.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f596bfff640 1 -- 172.21.15.62:0/2453463870 learned_addr learned my addr 172.21.15.62:0/2453463870 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:44:38.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f596bfff640 10 monclient(hunting): get_auth_request con 0x7f596c13b8a0 auth_method 0 2024-04-19T01:44:38.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f596bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:38.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f596bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:38.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f596bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:38.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f596affd640 10 monclient(hunting): get_auth_request con 0x7f596c09c990 auth_method 0 2024-04-19T01:44:38.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f596affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:38.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f596affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:38.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f596affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:38.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f596b7fe640 1 --2- 172.21.15.62:0/2453463870 >> v2:172.21.15.71:3300/0 conn(0x7f596c0a47e0 0x7f596c0a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:38.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f596b7fe640 10 monclient(hunting): get_auth_request con 0x7f596c0a47e0 auth_method 0 2024-04-19T01:44:38.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f596b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:38.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f596b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:38.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f596b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:38.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f596affd640 1 -- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c09c990 msgr2=0x7f596c0a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:44:38.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f596affd640 1 -- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c09c990 msgr2=0x7f596c0a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:38.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f596bfff640 1 -- 172.21.15.62:0/2453463870 >> v2:172.21.15.62:3300/0 conn(0x7f596c13b8a0 msgr2=0x7f596c13bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:44:38.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f596affd640 1 --2- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c09c990 0x7f596c0a5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:38.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f596bfff640 1 -- 172.21.15.62:0/2453463870 >> v2:172.21.15.62:3300/0 conn(0x7f596c13b8a0 msgr2=0x7f596c13bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:38.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f596bfff640 1 --2- 172.21.15.62:0/2453463870 >> v2:172.21.15.62:3300/0 conn(0x7f596c13b8a0 0x7f596c13bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:38.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f596affd640 1 --2- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c09c990 0x7f596c0a5100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:44:38.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f596bfff640 1 --2- 172.21.15.62:0/2453463870 >> v2:172.21.15.62:3300/0 conn(0x7f596c13b8a0 0x7f596c13bd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:44:38.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f596b7fe640 1 -- 172.21.15.62:0/2453463870 >> v2:172.21.15.71:3300/0 conn(0x7f596c0a47e0 msgr2=0x7f596c0a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:44:38.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f596b7fe640 1 -- 172.21.15.62:0/2453463870 >> v2:172.21.15.71:3300/0 conn(0x7f596c0a47e0 msgr2=0x7f596c0a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:38.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f596b7fe640 1 --2- 172.21.15.62:0/2453463870 >> v2:172.21.15.71:3300/0 conn(0x7f596c0a47e0 0x7f596c0a4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:38.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.421+0000 7f596b7fe640 1 --2- 172.21.15.62:0/2453463870 >> v2:172.21.15.71:3300/0 conn(0x7f596c0a47e0 0x7f596c0a4bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:44:38.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.625+0000 7f596affd640 1 --2- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c09c990 0x7f596c0a5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:38.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.625+0000 7f596bfff640 1 --2- 172.21.15.62:0/2453463870 >> v2:172.21.15.62:3300/0 conn(0x7f596c13b8a0 0x7f596c13bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:38.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.625+0000 7f596bfff640 10 monclient(hunting): get_auth_request con 0x7f596c13b8a0 auth_method 0 2024-04-19T01:44:38.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.625+0000 7f596bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:38.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.625+0000 7f596bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:38.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.625+0000 7f596bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:38.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.625+0000 7f596affd640 10 monclient(hunting): get_auth_request con 0x7f596c09c990 auth_method 0 2024-04-19T01:44:38.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.625+0000 7f596affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:38.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.625+0000 7f596affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:38.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.625+0000 7f596affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:38.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.625+0000 7f596b7fe640 1 --2- 172.21.15.62:0/2453463870 >> v2:172.21.15.71:3300/0 conn(0x7f596c0a47e0 0x7f596c0a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:38.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.625+0000 7f596bfff640 1 -- 172.21.15.62:0/2453463870 >> v2:172.21.15.62:3300/0 conn(0x7f596c13b8a0 msgr2=0x7f596c13bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:44:38.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.625+0000 7f596bfff640 1 -- 172.21.15.62:0/2453463870 >> v2:172.21.15.62:3300/0 conn(0x7f596c13b8a0 msgr2=0x7f596c13bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:38.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.625+0000 7f596bfff640 1 --2- 172.21.15.62:0/2453463870 >> v2:172.21.15.62:3300/0 conn(0x7f596c13b8a0 0x7f596c13bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:38.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.625+0000 7f596b7fe640 10 monclient(hunting): get_auth_request con 0x7f596c0a47e0 auth_method 0 2024-04-19T01:44:38.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.625+0000 7f596b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:38.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.625+0000 7f596b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:38.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.625+0000 7f596b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:38.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.625+0000 7f596affd640 1 -- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c09c990 msgr2=0x7f596c0a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:44:38.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.625+0000 7f596affd640 1 -- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c09c990 msgr2=0x7f596c0a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:38.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.625+0000 7f596affd640 1 --2- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c09c990 0x7f596c0a5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:38.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.625+0000 7f596bfff640 1 --2- 172.21.15.62:0/2453463870 >> v2:172.21.15.62:3300/0 conn(0x7f596c13b8a0 0x7f596c13bd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:44:38.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.625+0000 7f596affd640 1 --2- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c09c990 0x7f596c0a5100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:44:38.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.625+0000 7f596b7fe640 1 -- 172.21.15.62:0/2453463870 >> v2:172.21.15.71:3300/0 conn(0x7f596c0a47e0 msgr2=0x7f596c0a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:44:38.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.625+0000 7f596b7fe640 1 -- 172.21.15.62:0/2453463870 >> v2:172.21.15.71:3300/0 conn(0x7f596c0a47e0 msgr2=0x7f596c0a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:38.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.625+0000 7f596b7fe640 1 --2- 172.21.15.62:0/2453463870 >> v2:172.21.15.71:3300/0 conn(0x7f596c0a47e0 0x7f596c0a4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:38.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:38.625+0000 7f596b7fe640 1 --2- 172.21.15.62:0/2453463870 >> v2:172.21.15.71:3300/0 conn(0x7f596c0a47e0 0x7f596c0a4bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:44:39.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.029+0000 7f596affd640 1 --2- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c09c990 0x7f596c0a5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:39.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.029+0000 7f596bfff640 1 --2- 172.21.15.62:0/2453463870 >> v2:172.21.15.62:3300/0 conn(0x7f596c13b8a0 0x7f596c13bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:39.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.029+0000 7f596bfff640 10 monclient(hunting): get_auth_request con 0x7f596c13b8a0 auth_method 0 2024-04-19T01:44:39.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.029+0000 7f596bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:39.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.029+0000 7f596bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:39.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.029+0000 7f596bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:39.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.029+0000 7f596affd640 10 monclient(hunting): get_auth_request con 0x7f596c09c990 auth_method 0 2024-04-19T01:44:39.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.029+0000 7f596affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:39.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.029+0000 7f596affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:39.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.029+0000 7f596affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:39.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.029+0000 7f596b7fe640 1 --2- 172.21.15.62:0/2453463870 >> v2:172.21.15.71:3300/0 conn(0x7f596c0a47e0 0x7f596c0a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:39.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.029+0000 7f596b7fe640 10 monclient(hunting): get_auth_request con 0x7f596c0a47e0 auth_method 0 2024-04-19T01:44:39.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.029+0000 7f596b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:39.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.029+0000 7f596b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:39.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.029+0000 7f596b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:39.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.029+0000 7f596affd640 1 -- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c09c990 msgr2=0x7f596c0a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:44:39.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.029+0000 7f596affd640 1 -- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c09c990 msgr2=0x7f596c0a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:39.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.029+0000 7f596affd640 1 --2- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c09c990 0x7f596c0a5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:39.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.029+0000 7f596bfff640 1 -- 172.21.15.62:0/2453463870 >> v2:172.21.15.62:3300/0 conn(0x7f596c13b8a0 msgr2=0x7f596c13bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:44:39.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.029+0000 7f596bfff640 1 -- 172.21.15.62:0/2453463870 >> v2:172.21.15.62:3300/0 conn(0x7f596c13b8a0 msgr2=0x7f596c13bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:39.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.029+0000 7f596bfff640 1 --2- 172.21.15.62:0/2453463870 >> v2:172.21.15.62:3300/0 conn(0x7f596c13b8a0 0x7f596c13bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:39.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.029+0000 7f596b7fe640 1 -- 172.21.15.62:0/2453463870 >> v2:172.21.15.71:3300/0 conn(0x7f596c0a47e0 msgr2=0x7f596c0a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:44:39.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.029+0000 7f596b7fe640 1 -- 172.21.15.62:0/2453463870 >> v2:172.21.15.71:3300/0 conn(0x7f596c0a47e0 msgr2=0x7f596c0a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:39.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.029+0000 7f596b7fe640 1 --2- 172.21.15.62:0/2453463870 >> v2:172.21.15.71:3300/0 conn(0x7f596c0a47e0 0x7f596c0a4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:39.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.029+0000 7f596bfff640 1 --2- 172.21.15.62:0/2453463870 >> v2:172.21.15.62:3300/0 conn(0x7f596c13b8a0 0x7f596c13bd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:44:39.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.029+0000 7f596affd640 1 --2- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c09c990 0x7f596c0a5100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:44:39.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.029+0000 7f596b7fe640 1 --2- 172.21.15.62:0/2453463870 >> v2:172.21.15.71:3300/0 conn(0x7f596c0a47e0 0x7f596c0a4bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:44:39.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.833+0000 7f596bfff640 1 --2- 172.21.15.62:0/2453463870 >> v2:172.21.15.62:3300/0 conn(0x7f596c13b8a0 0x7f596c13bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:39.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.833+0000 7f596affd640 1 --2- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c09c990 0x7f596c0a5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:39.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.833+0000 7f596affd640 10 monclient(hunting): get_auth_request con 0x7f596c09c990 auth_method 0 2024-04-19T01:44:39.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.833+0000 7f596affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:39.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.833+0000 7f596affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:39.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.833+0000 7f596b7fe640 1 --2- 172.21.15.62:0/2453463870 >> v2:172.21.15.71:3300/0 conn(0x7f596c0a47e0 0x7f596c0a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:39.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.833+0000 7f596affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:39.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.833+0000 7f596bfff640 10 monclient(hunting): get_auth_request con 0x7f596c13b8a0 auth_method 0 2024-04-19T01:44:39.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.833+0000 7f596bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:39.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.833+0000 7f596bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:39.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.833+0000 7f596bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:39.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.833+0000 7f596b7fe640 10 monclient(hunting): get_auth_request con 0x7f596c0a47e0 auth_method 0 2024-04-19T01:44:39.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.833+0000 7f596b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:39.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.833+0000 7f596b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:39.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.833+0000 7f596b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:39.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.833+0000 7f596affd640 1 -- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c09c990 msgr2=0x7f596c0a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:44:39.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.833+0000 7f596affd640 1 -- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c09c990 msgr2=0x7f596c0a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:39.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.833+0000 7f596affd640 1 --2- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c09c990 0x7f596c0a5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:39.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.833+0000 7f596bfff640 1 -- 172.21.15.62:0/2453463870 >> v2:172.21.15.62:3300/0 conn(0x7f596c13b8a0 msgr2=0x7f596c13bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:44:39.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.833+0000 7f596bfff640 1 -- 172.21.15.62:0/2453463870 >> v2:172.21.15.62:3300/0 conn(0x7f596c13b8a0 msgr2=0x7f596c13bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:39.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.833+0000 7f596bfff640 1 --2- 172.21.15.62:0/2453463870 >> v2:172.21.15.62:3300/0 conn(0x7f596c13b8a0 0x7f596c13bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:39.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.833+0000 7f596b7fe640 1 -- 172.21.15.62:0/2453463870 >> v2:172.21.15.71:3300/0 conn(0x7f596c0a47e0 msgr2=0x7f596c0a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:44:39.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.833+0000 7f596b7fe640 1 -- 172.21.15.62:0/2453463870 >> v2:172.21.15.71:3300/0 conn(0x7f596c0a47e0 msgr2=0x7f596c0a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:39.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.833+0000 7f596affd640 1 --2- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c09c990 0x7f596c0a5100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:44:39.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.833+0000 7f596b7fe640 1 --2- 172.21.15.62:0/2453463870 >> v2:172.21.15.71:3300/0 conn(0x7f596c0a47e0 0x7f596c0a4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:39.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.833+0000 7f596bfff640 1 --2- 172.21.15.62:0/2453463870 >> v2:172.21.15.62:3300/0 conn(0x7f596c13b8a0 0x7f596c13bd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:44:39.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:39.833+0000 7f596b7fe640 1 --2- 172.21.15.62:0/2453463870 >> v2:172.21.15.71:3300/0 conn(0x7f596c0a47e0 0x7f596c0a4bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:44:41.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.421+0000 7f59697fa640 10 monclient(hunting): tick 2024-04-19T01:44:41.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.421+0000 7f59697fa640 10 monclient(hunting): _check_auth_tickets 2024-04-19T01:44:41.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.421+0000 7f59697fa640 1 monclient(hunting): continuing hunt 2024-04-19T01:44:41.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.421+0000 7f59697fa640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T01:44:41.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.421+0000 7f59697fa640 1 -- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c09c990 msgr2=0x7f596c0a5100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:44:41.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.421+0000 7f59697fa640 1 --2- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c09c990 0x7f596c0a5100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:41.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.421+0000 7f59697fa640 1 -- 172.21.15.62:0/2453463870 >> v2:172.21.15.71:3300/0 conn(0x7f596c0a47e0 msgr2=0x7f596c0a4bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:44:41.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.421+0000 7f59697fa640 1 --2- 172.21.15.62:0/2453463870 >> v2:172.21.15.71:3300/0 conn(0x7f596c0a47e0 0x7f596c0a4bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:41.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.421+0000 7f59697fa640 1 -- 172.21.15.62:0/2453463870 >> v2:172.21.15.62:3300/0 conn(0x7f596c13b8a0 msgr2=0x7f596c13bd30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:44:41.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.421+0000 7f59697fa640 1 --2- 172.21.15.62:0/2453463870 >> v2:172.21.15.62:3300/0 conn(0x7f596c13b8a0 0x7f596c13bd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:41.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.421+0000 7f59697fa640 10 monclient: _add_conns ranks=[1,0] 2024-04-19T01:44:41.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.421+0000 7f59697fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f59580036c0 addr v1:172.21.15.71:6789/0 2024-04-19T01:44:41.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.421+0000 7f59697fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f59580073d0 addr v1:172.21.15.62:6789/0 2024-04-19T01:44:41.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.421+0000 7f59697fa640 1 -- 172.21.15.62:0/2453463870 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f596c09de20 con 0x7f59580073d0 2024-04-19T01:44:41.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.421+0000 7f59697fa640 1 -- 172.21.15.62:0/2453463870 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f595800c0f0 con 0x7f59580036c0 2024-04-19T01:44:41.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.421+0000 7f59697fa640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:41.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596affd640 1 -- 172.21.15.62:0/2453463870 >> v1:172.21.15.62:6789/0 conn(0x7f59580073d0 legacy=0x7f5958009870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:41.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596b7fe640 1 -- 172.21.15.62:0/2453463870 >> v1:172.21.15.71:6789/0 conn(0x7f59580036c0 legacy=0x7f5958005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:41.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 1 -- 172.21.15.62:0/2453463870 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3343224917 0 0) 0x7f596c09de20 con 0x7f59580073d0 2024-04-19T01:44:41.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 10 monclient(hunting): _negotiate 2024-04-19T01:44:41.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:41.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:41.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 10 monclient(hunting): my global_id is 14124 2024-04-19T01:44:41.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 1 -- 172.21.15.62:0/2453463870 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f594c003530 con 0x7f59580073d0 2024-04-19T01:44:41.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 1 -- 172.21.15.62:0/2453463870 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3063703311 0 0) 0x7f595800c0f0 con 0x7f59580036c0 2024-04-19T01:44:41.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 10 monclient(hunting): _negotiate 2024-04-19T01:44:41.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:41.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:41.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 10 monclient(hunting): my global_id is 4690 2024-04-19T01:44:41.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 1 -- 172.21.15.62:0/2453463870 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f596c09de20 con 0x7f59580036c0 2024-04-19T01:44:41.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 1 -- 172.21.15.62:0/2453463870 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3900424930 0 0) 0x7f594c003530 con 0x7f59580073d0 2024-04-19T01:44:41.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 1 -- 172.21.15.62:0/2453463870 >> v1:172.21.15.71:6789/0 conn(0x7f59580036c0 legacy=0x7f5958005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:41.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 10 monclient: _finish_hunting 0 2024-04-19T01:44:41.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 1 monclient: found mon.noname-a-legacy 2024-04-19T01:44:41.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:41.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.62:6789/0 2024-04-19T01:44:41.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 1 -- 172.21.15.62:0/2453463870 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f595800d2f0 con 0x7f59580073d0 2024-04-19T01:44:41.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 10 monclient: _finish_auth 0 2024-04-19T01:44:41.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 10 monclient: _check_auth_tickets 2024-04-19T01:44:41.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:41.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 1 -- 172.21.15.62:0/2453463870 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1244427781 0 0) 0x7f595400f040 con 0x7f59580073d0 2024-04-19T01:44:41.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:41.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-19T01:44:41.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 10 monclient: dump: 2024-04-19T01:44:41.426 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:41.427 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:41.427 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:41.427 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:41.427 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:41.427 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:41.427 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:41.427 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:41.427 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:41.427 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:41.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] but i'm connected to v1:172.21.15.62:6789/0, reconnecting 2024-04-19T01:44:41.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:41.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 1 -- 172.21.15.62:0/2453463870 >> v1:172.21.15.62:6789/0 conn(0x7f59580073d0 legacy=0x7f5958009870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:41.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-19T01:44:41.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596bfff640 1 -- 172.21.15.62:0/2453463870 reap_dead start 2024-04-19T01:44:41.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 1 --2- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f594c004d50 0x7f594c005130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:41.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 10 monclient(hunting): picked mon.a con 0x7f594c004d50 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:44:41.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 1 --2- 172.21.15.62:0/2453463870 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f594c003800 0x7f594c003c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:41.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 10 monclient(hunting): picked mon.b con 0x7f594c003800 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:44:41.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 1 --2- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f594c0099c0 0x7f594c00bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:41.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 10 monclient(hunting): picked mon.c con 0x7f594c0099c0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:44:41.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:41.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 1 -- 172.21.15.62:0/2453463870 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f594c00c480 con 0x7f594c004d50 2024-04-19T01:44:41.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:41.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 1 -- 172.21.15.62:0/2453463870 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f594c00c600 con 0x7f594c003800 2024-04-19T01:44:41.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:41.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 1 -- 172.21.15.62:0/2453463870 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f594c00c800 con 0x7f594c0099c0 2024-04-19T01:44:41.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:41.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596a7fc640 10 monclient(hunting): _finish_auth 0 2024-04-19T01:44:41.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596b7fe640 1 --2- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f594c004d50 0x7f594c005130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:41.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596bfff640 1 --2- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f594c0099c0 0x7f594c00bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:41.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596b7fe640 10 monclient(hunting): get_auth_request con 0x7f594c004d50 auth_method 0 2024-04-19T01:44:41.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:41.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:41.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:41.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596bfff640 10 monclient(hunting): get_auth_request con 0x7f594c0099c0 auth_method 0 2024-04-19T01:44:41.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:41.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:41.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:41.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596affd640 1 --2- 172.21.15.62:0/2453463870 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f594c003800 0x7f594c003c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:41.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596affd640 10 monclient(hunting): get_auth_request con 0x7f594c003800 auth_method 0 2024-04-19T01:44:41.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:41.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:41.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:41.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:41.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:41.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T01:44:41.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:41.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:41.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T01:44:41.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:41.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:41.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.425+0000 7f596affd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T01:44:41.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.429+0000 7f596b7fe640 10 monclient(hunting): handle_auth_done global_id 14124 payload 293 2024-04-19T01:44:41.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.429+0000 7f596b7fe640 1 -- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f594c0099c0 msgr2=0x7f594c00bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:44:41.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.429+0000 7f596b7fe640 1 --2- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f594c0099c0 0x7f594c00bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:41.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.429+0000 7f596b7fe640 1 -- 172.21.15.62:0/2453463870 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f594c003800 msgr2=0x7f594c003c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:44:41.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.429+0000 7f596b7fe640 1 --2- 172.21.15.62:0/2453463870 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f594c003800 0x7f594c003c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:41.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.429+0000 7f596b7fe640 10 monclient: _finish_hunting 0 2024-04-19T01:44:41.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.429+0000 7f596b7fe640 1 monclient: found mon.a 2024-04-19T01:44:41.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.429+0000 7f596b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:41.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.429+0000 7f596b7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:44:41.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.429+0000 7f596b7fe640 1 -- 172.21.15.62:0/2453463870 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f594c003530 con 0x7f594c004d50 2024-04-19T01:44:41.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.429+0000 7f596b7fe640 10 monclient: _finish_auth 0 2024-04-19T01:44:41.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.429+0000 7f596b7fe640 10 monclient: _check_auth_tickets 2024-04-19T01:44:41.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.429+0000 7f596b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:41.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.429+0000 7f5971edb640 5 monclient: authenticate success, global_id 14124 2024-04-19T01:44:41.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.429+0000 7f5971edb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:44:41.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.437+0000 7f596b7fe640 1 -- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f594c004d50 msgr2=0x7f594c005130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:41.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.437+0000 7f596b7fe640 1 --2- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f594c004d50 0x7f594c005130 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f595c009610 tx=0x7f595c01db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee1b632857ac2995 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:41.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.437+0000 7f596a7fc640 1 -- 172.21.15.62:0/2453463870 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f595c0ac070 con 0x7f594c004d50 2024-04-19T01:44:41.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.437+0000 7f596a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:41.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.437+0000 7f596a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:44:41.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.437+0000 7f596a7fc640 10 monclient: dump: 2024-04-19T01:44:41.433 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:41.433 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:41.433 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:41.434 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:41.434 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:41.434 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:41.434 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:41.434 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:41.434 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:41.434 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:41.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.437+0000 7f596a7fc640 1 -- 172.21.15.62:0/2453463870 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f595c0a6b30 con 0x7f594c004d50 2024-04-19T01:44:41.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.437+0000 7f5971edb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:44:41.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.437+0000 7f596a7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:41.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.437+0000 7f5971edb640 10 monclient: get_monmap_and_config success 2024-04-19T01:44:41.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.437+0000 7f596a7fc640 1 -- 172.21.15.62:0/2453463870 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f595c004040 con 0x7f594c004d50 2024-04-19T01:44:41.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.437+0000 7f596a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:41.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.437+0000 7f5971edb640 4 set_mon_vals no callback set 2024-04-19T01:44:41.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.437+0000 7f596a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:44:41.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.437+0000 7f596a7fc640 10 monclient: dump: 2024-04-19T01:44:41.435 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:41.435 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:41.435 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:41.435 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:41.435 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:41.435 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:41.435 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:41.435 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:41.435 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:41.435 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:41.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.437+0000 7f5971edb640 10 monclient: shutdown 2024-04-19T01:44:41.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.437+0000 7f5971edb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:41.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.437+0000 7f5971edb640 1 -- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f594c004d50 msgr2=0x7f594c005130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:41.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.437+0000 7f5971edb640 1 --2- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f594c004d50 0x7f594c005130 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f595c009610 tx=0x7f595c01db90 comp rx=0 tx=0).stop 2024-04-19T01:44:41.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.437+0000 7f5971edb640 1 -- 172.21.15.62:0/2453463870 shutdown_connections 2024-04-19T01:44:41.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.437+0000 7f5971edb640 1 --2- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f594c0099c0 0x7f594c00bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:41.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.437+0000 7f5971edb640 1 --2- 172.21.15.62:0/2453463870 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f594c003800 0x7f594c003c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:41.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.437+0000 7f5971edb640 1 --2- 172.21.15.62:0/2453463870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f594c004d50 0x7f594c005130 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:41.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.437+0000 7f5971edb640 1 -- 172.21.15.62:0/2453463870 >> 172.21.15.62:0/2453463870 conn(0x7f596c0a23d0 msgr2=0x7f596c0a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:41.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.437+0000 7f5971edb640 1 -- 172.21.15.62:0/2453463870 shutdown_connections 2024-04-19T01:44:41.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.437+0000 7f5971edb640 1 -- 172.21.15.62:0/2453463870 wait complete. 2024-04-19T01:44:41.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.437+0000 7f5971edb640 10 monclient: build_initial_monmap 2024-04-19T01:44:41.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.437+0000 7f5971edb640 10 monclient: monmap: 2024-04-19T01:44:41.437 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:41.437 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:44:41.437 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:44:41.437 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:44:41.437 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:41.437 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:41.437 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:44:41.437 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:44:41.437 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:44:41.438 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:41.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f5971edb640 1 Processor -- start 2024-04-19T01:44:41.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f5971edb640 1 -- start start 2024-04-19T01:44:41.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f5971edb640 10 monclient: init 2024-04-19T01:44:41.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f5971edb640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:41.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f5971edb640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-19T01:44:41.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f5971edb640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f594c003800 0x7f596c1d32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:41.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f5971edb640 10 monclient(hunting): picked mon.noname-a con 0x7f594c003800 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:44:41.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f5971edb640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f594c0099c0 0x7f596c13aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:41.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f5971edb640 10 monclient(hunting): picked mon.noname-b con 0x7f594c0099c0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:44:41.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f5971edb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c13af80 0x7f596c124730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:41.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f5971edb640 10 monclient(hunting): picked mon.noname-c con 0x7f596c13af80 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:44:41.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f5971edb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:41.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f5971edb640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f596c13bcb0 con 0x7f594c003800 2024-04-19T01:44:41.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f5971edb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:41.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f5971edb640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f596c09e8e0 con 0x7f594c0099c0 2024-04-19T01:44:41.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f5971edb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:41.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f5971edb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f596c13be30 con 0x7f596c13af80 2024-04-19T01:44:41.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f5971edb640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:41.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f5971edb640 10 monclient(hunting): authenticate will time out at 1067.261987s 2024-04-19T01:44:41.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f594c003800 0x7f596c1d32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:41.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f594c003800 0x7f596c1d32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44506/0 (socket says 172.21.15.62:44506) 2024-04-19T01:44:41.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596b7fe640 1 -- 172.21.15.62:0/3969261861 learned_addr learned my addr 172.21.15.62:0/3969261861 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:44:41.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596b7fe640 10 monclient(hunting): get_auth_request con 0x7f594c003800 auth_method 0 2024-04-19T01:44:41.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:41.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:41.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:41.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596bfff640 1 --2- 172.21.15.62:0/3969261861 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c13af80 0x7f596c124730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:41.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596bfff640 10 monclient(hunting): get_auth_request con 0x7f596c13af80 auth_method 0 2024-04-19T01:44:41.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:41.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:41.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:41.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596b7fe640 1 -- 172.21.15.62:0/3969261861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f594c003800 msgr2=0x7f596c1d32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-19T01:44:41.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596b7fe640 1 -- 172.21.15.62:0/3969261861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f594c003800 msgr2=0x7f596c1d32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:44:41.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596b7fe640 1 --2- 172.21.15.62:0/3969261861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f594c003800 0x7f596c1d32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:41.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596b7fe640 1 --2- 172.21.15.62:0/3969261861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f594c003800 0x7f596c1d32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:44:41.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596affd640 1 --2- 172.21.15.62:0/3969261861 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f594c0099c0 0x7f596c13aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:41.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596affd640 10 monclient(hunting): get_auth_request con 0x7f594c0099c0 auth_method 0 2024-04-19T01:44:41.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:41.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:41.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:41.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596bfff640 1 -- 172.21.15.62:0/3969261861 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c13af80 msgr2=0x7f596c124730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-19T01:44:41.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596bfff640 1 -- 172.21.15.62:0/3969261861 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c13af80 msgr2=0x7f596c124730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:44:41.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596bfff640 1 --2- 172.21.15.62:0/3969261861 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c13af80 0x7f596c124730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:41.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:41.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596bfff640 1 --2- 172.21.15.62:0/3969261861 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c13af80 0x7f596c124730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:44:41.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:41.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:41.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596affd640 10 monclient(hunting): handle_auth_done global_id 4693 payload 824 2024-04-19T01:44:41.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596affd640 1 -- 172.21.15.62:0/3969261861 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c13af80 msgr2=0x7f596c124730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:44:41.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596affd640 1 --2- 172.21.15.62:0/3969261861 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c13af80 0x7f596c124730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:41.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596affd640 1 -- 172.21.15.62:0/3969261861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f594c003800 msgr2=0x7f596c1d32b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:44:41.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596affd640 1 --2- 172.21.15.62:0/3969261861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f594c003800 0x7f596c1d32b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:41.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596affd640 10 monclient: _finish_hunting 0 2024-04-19T01:44:41.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596affd640 1 monclient: found mon.noname-b 2024-04-19T01:44:41.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:41.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.71:3300/0 2024-04-19T01:44:41.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.441+0000 7f596affd640 1 -- 172.21.15.62:0/3969261861 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f596c128400 con 0x7f594c0099c0 2024-04-19T01:44:41.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.445+0000 7f596affd640 1 -- 172.21.15.62:0/3969261861 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f594c0099c0 msgr2=0x7f596c13aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:41.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.445+0000 7f596affd640 1 --2- 172.21.15.62:0/3969261861 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f594c0099c0 0x7f596c13aa40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5954000ee0 tx=0x7f59540146c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:41.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.445+0000 7f5963fff640 1 -- 172.21.15.62:0/3969261861 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5954004020 con 0x7f594c0099c0 2024-04-19T01:44:41.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.445+0000 7f5963fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:41.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.445+0000 7f5963fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:44:41.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.445+0000 7f5963fff640 10 monclient: dump: 2024-04-19T01:44:41.446 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:41.446 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:41.446 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:41.446 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:41.446 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:41.446 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:41.446 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:41.446 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:41.447 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:41.447 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:41.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.445+0000 7f5963fff640 10 monclient: _finish_auth 0 2024-04-19T01:44:41.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.445+0000 7f5963fff640 10 monclient: _check_auth_tickets 2024-04-19T01:44:41.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.445+0000 7f5963fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:41.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.445+0000 7f5971edb640 5 monclient: authenticate success, global_id 4693 2024-04-19T01:44:41.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.445+0000 7f5971edb640 10 monclient: _renew_subs 2024-04-19T01:44:41.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.445+0000 7f5971edb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:44:41.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.445+0000 7f5971edb640 1 -- 172.21.15.62:0/3969261861 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f596c072240 con 0x7f594c0099c0 2024-04-19T01:44:41.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.445+0000 7f5963fff640 1 -- 172.21.15.62:0/3969261861 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59540041e0 con 0x7f594c0099c0 2024-04-19T01:44:41.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.445+0000 7f5963fff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:41.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.445+0000 7f5963fff640 1 -- 172.21.15.62:0/3969261861 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5954018870 con 0x7f594c0099c0 2024-04-19T01:44:41.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.445+0000 7f5963fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:41.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.445+0000 7f5963fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:44:41.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.445+0000 7f5963fff640 10 monclient: dump: 2024-04-19T01:44:41.448 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:41.448 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:41.448 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:41.448 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:41.448 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:41.448 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:41.448 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:41.448 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:41.448 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:41.448 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:41.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.445+0000 7f5969ffb640 4 set_mon_vals no callback set 2024-04-19T01:44:41.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.445+0000 7f5971edb640 10 monclient: _renew_subs 2024-04-19T01:44:41.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.445+0000 7f5971edb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:44:41.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.445+0000 7f5971edb640 1 -- 172.21.15.62:0/3969261861 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f596c128120 con 0x7f594c0099c0 2024-04-19T01:44:41.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.445+0000 7f5971edb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:41.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.445+0000 7f5971edb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:41.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.445+0000 7f5971edb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:44:41.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.445+0000 7f5971edb640 1 -- 172.21.15.62:0/3969261861 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f596c0692b0 con 0x7f594c0099c0 2024-04-19T01:44:41.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.445+0000 7f5963fff640 1 -- 172.21.15.62:0/3969261861 <== mon.1 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f5954020030 con 0x7f594c0099c0 2024-04-19T01:44:41.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.445+0000 7f5963fff640 1 --2- 172.21.15.62:0/3969261861 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5958036ca0 0x7f5958039160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:41.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.445+0000 7f5963fff640 1 -- 172.21.15.62:0/3969261861 <== mon.1 v2:172.21.15.71:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 4738+0+0 (secure 0 0 0) 0x7f595404a220 con 0x7f594c0099c0 2024-04-19T01:44:41.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.445+0000 7f596b7fe640 1 --2- 172.21.15.62:0/3969261861 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5958036ca0 0x7f5958039160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:41.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.449+0000 7f596b7fe640 10 monclient: get_auth_request con 0x7f5958036ca0 auth_method 0 2024-04-19T01:44:41.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.449+0000 7f596b7fe640 1 --2- 172.21.15.62:0/3969261861 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5958036ca0 0x7f5958039160 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f595c009790 tx=0x7f595c004ac0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:41.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.461+0000 7f5963fff640 1 -- 172.21.15.62:0/3969261861 <== mon.1 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f59540039a0 con 0x7f594c0099c0 2024-04-19T01:44:41.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.461+0000 7f5963fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:41.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.461+0000 7f5963fff640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:44:41.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.689+0000 7f5971edb640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:44:41.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.689+0000 7f5971edb640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:44:41.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.689+0000 7f5971edb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:44:41.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.689+0000 7f5971edb640 1 -- 172.21.15.62:0/3969261861 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f596c126980 con 0x7f594c0099c0 2024-04-19T01:44:41.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.689+0000 7f5963fff640 1 -- 172.21.15.62:0/3969261861 <== mon.1 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f595402ac50 con 0x7f594c0099c0 2024-04-19T01:44:41.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.689+0000 7f5963fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:44:41.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.689+0000 7f5963fff640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:44:41.688 INFO:teuthology.orchestra.run.smithi062.stdout:{"election_epoch":26,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","modified":"2024-04-19T01:41:38.497795Z","created":"2024-04-19T01:41:38.497795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:44:41.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.693+0000 7f5971edb640 1 -- 172.21.15.62:0/3969261861 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5958036ca0 msgr2=0x7f5958039160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:41.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.693+0000 7f5971edb640 1 --2- 172.21.15.62:0/3969261861 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5958036ca0 0x7f5958039160 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f595c009790 tx=0x7f595c004ac0 comp rx=0 tx=0).stop 2024-04-19T01:44:41.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.693+0000 7f5971edb640 10 monclient: shutdown 2024-04-19T01:44:41.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.693+0000 7f5971edb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:41.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.693+0000 7f5971edb640 1 -- 172.21.15.62:0/3969261861 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f594c0099c0 msgr2=0x7f596c13aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:41.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.693+0000 7f5971edb640 1 --2- 172.21.15.62:0/3969261861 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f594c0099c0 0x7f596c13aa40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5954000ee0 tx=0x7f59540146c0 comp rx=0 tx=0).stop 2024-04-19T01:44:41.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.693+0000 7f5971edb640 1 -- 172.21.15.62:0/3969261861 shutdown_connections 2024-04-19T01:44:41.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.693+0000 7f5971edb640 1 --2- 172.21.15.62:0/3969261861 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5958036ca0 0x7f5958039160 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:41.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.693+0000 7f5971edb640 1 --2- 172.21.15.62:0/3969261861 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f596c13af80 0x7f596c124730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:41.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.693+0000 7f5971edb640 1 --2- 172.21.15.62:0/3969261861 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f594c0099c0 0x7f596c13aa40 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:41.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.693+0000 7f5971edb640 1 --2- 172.21.15.62:0/3969261861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f594c003800 0x7f596c1d32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:41.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.693+0000 7f5971edb640 1 -- 172.21.15.62:0/3969261861 >> 172.21.15.62:0/3969261861 conn(0x7f596c0a23d0 msgr2=0x7f596c0a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:41.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.693+0000 7f5971edb640 1 -- 172.21.15.62:0/3969261861 shutdown_connections 2024-04-19T01:44:41.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:41.693+0000 7f5971edb640 1 -- 172.21.15.62:0/3969261861 wait complete. 2024-04-19T01:44:42.523 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:44:42.523 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28820]: starting. 2024-04-19T01:44:42.523 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28820]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-28783 2024-04-19T01:44:42.523 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28820]: created object 0... 2024-04-19T01:44:42.523 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28820]: created object 25... 2024-04-19T01:44:42.523 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28820]: created object 49... 2024-04-19T01:44:42.523 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28820]: finishing. 2024-04-19T01:44:42.523 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28820]: shutting down. 2024-04-19T01:44:42.584 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:44:42.584 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28822]: starting. 2024-04-19T01:44:42.584 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28822]: listing objects. 2024-04-19T01:44:42.585 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28822]: listed object 0... 2024-04-19T01:44:42.585 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28822]: listed object 25... 2024-04-19T01:44:42.585 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28822]: saw 50 objects 2024-04-19T01:44:42.585 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28822]: shutting down. 2024-04-19T01:44:43.110 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:44:43.110 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28821]: starting. 2024-04-19T01:44:43.110 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28821]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-28783 2024-04-19T01:44:43.110 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28821]: shutting down. 2024-04-19T01:44:43.113 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:44:43.113 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:44:43.113 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:44:43.116 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:44:44.708 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:44:44.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.845+0000 7f8899324640 10 monclient: get_monmap_and_config 2024-04-19T01:44:44.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.845+0000 7f8899324640 10 monclient: build_initial_monmap 2024-04-19T01:44:44.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.845+0000 7f8899324640 10 monclient: monmap: 2024-04-19T01:44:44.843 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:44.843 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:44.843 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:44:44.848455+0000 2024-04-19T01:44:44.843 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:44:44.848455+0000 2024-04-19T01:44:44.843 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:44.843 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:44.844 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:44:44.844 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:44:44.844 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:44:44.844 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:44:44.844 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:44:44.844 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:44.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.845+0000 7f8899324640 1 Processor -- start 2024-04-19T01:44:44.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.845+0000 7f8899324640 1 -- start start 2024-04-19T01:44:44.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.845+0000 7f8899324640 10 monclient: init 2024-04-19T01:44:44.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f8899324640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:44.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f8899324640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-04-19T01:44:44.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f8899324640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f88940a3410 addr v1:172.21.15.62:6789/0 2024-04-19T01:44:44.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f8899324640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f88940a2900 0x7f889413c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:44.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f8899324640 10 monclient(hunting): picked mon.noname-c con 0x7f88940a2900 addr v2:172.21.15.71:3300/0 2024-04-19T01:44:44.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f8899324640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f88940a2d50 0x7f889413ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:44.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f8899324640 10 monclient(hunting): picked mon.noname-a con 0x7f88940a2d50 addr v2:172.21.15.62:3300/0 2024-04-19T01:44:44.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f8899324640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88940a56e0 con 0x7f88940a3410 2024-04-19T01:44:44.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f8899324640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:44.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f8899324640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f88940a1430 con 0x7f88940a2d50 2024-04-19T01:44:44.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f8899324640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:44.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f8899324640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f88940a1710 con 0x7f88940a2900 2024-04-19T01:44:44.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f8899324640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:44.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f8899324640 10 monclient(hunting): authenticate will time out at 1070.668676s 2024-04-19T01:44:44.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f88937fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f88940a2d50 0x7f889413ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:44.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f8892ffd640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f88940a3410 0x7f88940a37f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:48138/0 (socket says 172.21.15.62:48138) 2024-04-19T01:44:44.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f8892ffd640 1 -- 172.21.15.62:0/421576594 learned_addr learned my addr 172.21.15.62:0/421576594 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:44:44.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f88937fe640 10 monclient(hunting): get_auth_request con 0x7f88940a2d50 auth_method 0 2024-04-19T01:44:44.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f88937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:44.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f88937fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:44.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f88937fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:44.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f88937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:44.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f88937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:44.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f88937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:44.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f888bfff640 1 --2- 172.21.15.62:0/421576594 >> v2:172.21.15.71:3300/0 conn(0x7f88940a2900 0x7f889413c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:44.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f888bfff640 10 monclient(hunting): get_auth_request con 0x7f88940a2900 auth_method 0 2024-04-19T01:44:44.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f888bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:44.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f888bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:44.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f888bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:44.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f88937fe640 10 monclient(hunting): handle_auth_done global_id 14142 payload 274 2024-04-19T01:44:44.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f88937fe640 1 -- 172.21.15.62:0/421576594 >> v2:172.21.15.71:3300/0 conn(0x7f88940a2900 msgr2=0x7f889413c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:44:44.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f88937fe640 1 --2- 172.21.15.62:0/421576594 >> v2:172.21.15.71:3300/0 conn(0x7f88940a2900 0x7f889413c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:44.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f88937fe640 1 -- 172.21.15.62:0/421576594 >> v1:172.21.15.62:6789/0 conn(0x7f88940a3410 legacy=0x7f88940a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:44:44.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f88937fe640 10 monclient: _finish_hunting 0 2024-04-19T01:44:44.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f88937fe640 1 monclient: found mon.noname-a 2024-04-19T01:44:44.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f88937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:44.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f88937fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:44:44.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.849+0000 7f88937fe640 1 -- 172.21.15.62:0/421576594 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88940a19f0 con 0x7f88940a2d50 2024-04-19T01:44:44.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f88937fe640 1 -- 172.21.15.62:0/421576594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88940a2d50 msgr2=0x7f889413ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:44.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f88937fe640 1 --2- 172.21.15.62:0/421576594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88940a2d50 0x7f889413ed00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f88940738e0 tx=0x7f888c01c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce434e3f10823f46 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:44.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f88927fc640 1 -- 172.21.15.62:0/421576594 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888c0b2070 con 0x7f88940a2d50 2024-04-19T01:44:44.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f88927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:44.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f88927fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:44:44.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f88927fc640 10 monclient: dump: 2024-04-19T01:44:44.851 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:44.851 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:44.851 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:44.851 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:44.851 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:44.851 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:44.851 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:44.851 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:44.851 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:44.851 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:44.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f88927fc640 10 monclient: _finish_auth 0 2024-04-19T01:44:44.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f88927fc640 10 monclient: _check_auth_tickets 2024-04-19T01:44:44.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f88927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:44.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f88927fc640 1 -- 172.21.15.62:0/421576594 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f888c0a5870 con 0x7f88940a2d50 2024-04-19T01:44:44.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f8899324640 5 monclient: authenticate success, global_id 14142 2024-04-19T01:44:44.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f8899324640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:44:44.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f88927fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:44.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f8899324640 10 monclient: get_monmap_and_config success 2024-04-19T01:44:44.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f8899324640 4 set_mon_vals no callback set 2024-04-19T01:44:44.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f88927fc640 1 -- 172.21.15.62:0/421576594 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888c0a5c40 con 0x7f88940a2d50 2024-04-19T01:44:44.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f88927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:44.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f88927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:44:44.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f88927fc640 10 monclient: dump: 2024-04-19T01:44:44.852 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:44.852 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:44.852 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:44.852 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:44.852 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:44.852 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:44.852 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:44.852 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:44.853 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:44.853 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:44.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f8899324640 10 monclient: shutdown 2024-04-19T01:44:44.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f8899324640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:44.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f8899324640 1 -- 172.21.15.62:0/421576594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88940a2d50 msgr2=0x7f889413ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:44.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f8899324640 1 --2- 172.21.15.62:0/421576594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88940a2d50 0x7f889413ed00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f88940738e0 tx=0x7f888c01c7d0 comp rx=0 tx=0).stop 2024-04-19T01:44:44.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f8899324640 1 -- 172.21.15.62:0/421576594 shutdown_connections 2024-04-19T01:44:44.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f8899324640 1 --2- 172.21.15.62:0/421576594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88940a2d50 0x7f889413ed00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:44.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f8899324640 1 --2- 172.21.15.62:0/421576594 >> v2:172.21.15.71:3300/0 conn(0x7f88940a2900 0x7f889413c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:44.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f8899324640 1 -- 172.21.15.62:0/421576594 >> 172.21.15.62:0/421576594 conn(0x7f88940aa2d0 msgr2=0x7f88940a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:44.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f8899324640 1 -- 172.21.15.62:0/421576594 shutdown_connections 2024-04-19T01:44:44.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f8899324640 1 -- 172.21.15.62:0/421576594 wait complete. 2024-04-19T01:44:44.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f8899324640 10 monclient: build_initial_monmap 2024-04-19T01:44:44.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f8899324640 10 monclient: monmap: 2024-04-19T01:44:44.856 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:44.856 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:44:44.856 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:44:44.856 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:44:44.856 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:44.856 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:44.856 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:44:44.857 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:44:44.857 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:44:44.857 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:44.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f8899324640 1 Processor -- start 2024-04-19T01:44:44.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f8899324640 1 -- start start 2024-04-19T01:44:44.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.853+0000 7f8899324640 10 monclient: init 2024-04-19T01:44:44.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8899324640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:44.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8899324640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-19T01:44:44.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8899324640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88940a2900 0x7f88941cf160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:44.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8899324640 10 monclient(hunting): picked mon.noname-c con 0x7f88940a2900 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:44:44.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8899324640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88940a2d50 0x7f889413d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:44.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8899324640 10 monclient(hunting): picked mon.noname-b con 0x7f88940a2d50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:44:44.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8899324640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88940a3410 0x7f8894124070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:44.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8899324640 10 monclient(hunting): picked mon.noname-a con 0x7f88940a3410 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:44:44.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8899324640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:44.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8899324640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8894141df0 con 0x7f88940a3410 2024-04-19T01:44:44.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8899324640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:44.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8899324640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8894141c70 con 0x7f88940a2d50 2024-04-19T01:44:44.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8899324640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:44.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8899324640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f8894141f70 con 0x7f88940a2900 2024-04-19T01:44:44.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8899324640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:44.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8899324640 10 monclient(hunting): authenticate will time out at 1070.676431s 2024-04-19T01:44:44.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f88937fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88940a3410 0x7f8894124070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:44.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8892ffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88940a2900 0x7f88941cf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:44.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f88937fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88940a3410 0x7f8894124070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44528/0 (socket says 172.21.15.62:44528) 2024-04-19T01:44:44.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f88937fe640 1 -- 172.21.15.62:0/1300565019 learned_addr learned my addr 172.21.15.62:0/1300565019 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:44:44.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f88937fe640 10 monclient(hunting): get_auth_request con 0x7f88940a3410 auth_method 0 2024-04-19T01:44:44.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f88937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:44.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f88937fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:44.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f88937fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:44.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8892ffd640 10 monclient(hunting): get_auth_request con 0x7f88940a2900 auth_method 0 2024-04-19T01:44:44.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8892ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:44.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8892ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:44.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8892ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:44.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8892ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:44.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8892ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:44.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8892ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:44.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f88937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:44.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f88937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:44.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f88937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:44.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f888bfff640 1 --2- 172.21.15.62:0/1300565019 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88940a2d50 0x7f889413d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:44.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f888bfff640 10 monclient(hunting): get_auth_request con 0x7f88940a2d50 auth_method 0 2024-04-19T01:44:44.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f888bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:44.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f888bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:44.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f888bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:44.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8892ffd640 10 monclient(hunting): handle_auth_done global_id 4745 payload 824 2024-04-19T01:44:44.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8892ffd640 1 -- 172.21.15.62:0/1300565019 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88940a2d50 msgr2=0x7f889413d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:44.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8892ffd640 1 --2- 172.21.15.62:0/1300565019 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88940a2d50 0x7f889413d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:44.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8892ffd640 1 -- 172.21.15.62:0/1300565019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88940a3410 msgr2=0x7f8894124070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:44.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8892ffd640 1 --2- 172.21.15.62:0/1300565019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88940a3410 0x7f8894124070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:44.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8892ffd640 10 monclient: _finish_hunting 0 2024-04-19T01:44:44.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8892ffd640 1 monclient: found mon.noname-c 2024-04-19T01:44:44.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8892ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:44.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8892ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:44:44.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8892ffd640 1 -- 172.21.15.62:0/1300565019 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f889413e790 con 0x7f88940a2900 2024-04-19T01:44:44.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8892ffd640 1 -- 172.21.15.62:0/1300565019 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88940a2900 msgr2=0x7f88941cf160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:44.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8892ffd640 1 --2- 172.21.15.62:0/1300565019 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88940a2900 0x7f88941cf160 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f887c00c440 tx=0x7f887c00c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:44.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f888b7fe640 1 -- 172.21.15.62:0/1300565019 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f887c017070 con 0x7f88940a2900 2024-04-19T01:44:44.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f888b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:44.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f888b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:44:44.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f888b7fe640 10 monclient: dump: 2024-04-19T01:44:44.862 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:44.862 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:44.862 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:44.863 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:44.863 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:44.863 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:44.863 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:44.863 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:44.863 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:44.863 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:44.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f888b7fe640 10 monclient: _finish_auth 0 2024-04-19T01:44:44.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f888b7fe640 10 monclient: _check_auth_tickets 2024-04-19T01:44:44.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f888b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:44.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8899324640 5 monclient: authenticate success, global_id 4745 2024-04-19T01:44:44.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8899324640 10 monclient: _renew_subs 2024-04-19T01:44:44.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8899324640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:44:44.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8899324640 1 -- 172.21.15.62:0/1300565019 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8894125c70 con 0x7f88940a2900 2024-04-19T01:44:44.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f888b7fe640 1 -- 172.21.15.62:0/1300565019 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f887c0070d0 con 0x7f88940a2900 2024-04-19T01:44:44.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f888b7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:44.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f888b7fe640 1 -- 172.21.15.62:0/1300565019 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f887c0074c0 con 0x7f88940a2900 2024-04-19T01:44:44.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f888b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:44.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f888b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:44:44.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f888b7fe640 10 monclient: dump: 2024-04-19T01:44:44.864 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:44.864 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:44.864 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:44.864 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:44.864 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:44.864 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:44.864 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:44.864 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:44.864 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:44.864 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:44.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8891ffb640 4 set_mon_vals no callback set 2024-04-19T01:44:44.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8899324640 10 monclient: _renew_subs 2024-04-19T01:44:44.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8899324640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:44:44.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.857+0000 7f8899324640 1 -- 172.21.15.62:0/1300565019 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f88941cffe0 con 0x7f88940a2900 2024-04-19T01:44:44.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.861+0000 7f8899324640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:44.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.861+0000 7f8899324640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:44.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.861+0000 7f8899324640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:44:44.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.861+0000 7f8899324640 1 -- 172.21.15.62:0/1300565019 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88940692b0 con 0x7f88940a2900 2024-04-19T01:44:44.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.861+0000 7f888b7fe640 1 -- 172.21.15.62:0/1300565019 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f887c01a070 con 0x7f88940a2900 2024-04-19T01:44:44.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.861+0000 7f888b7fe640 1 --2- 172.21.15.62:0/1300565019 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f8874036420 0x7f88740388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:44.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.861+0000 7f888b7fe640 1 -- 172.21.15.62:0/1300565019 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(114..114 src has 1..114) ==== 4738+0+0 (secure 0 0 0) 0x7f887c048840 con 0x7f88940a2900 2024-04-19T01:44:44.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.861+0000 7f888bfff640 1 --2- 172.21.15.62:0/1300565019 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f8874036420 0x7f88740388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:44.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.861+0000 7f888bfff640 10 monclient: get_auth_request con 0x7f8874036420 auth_method 0 2024-04-19T01:44:44.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.865+0000 7f888bfff640 1 --2- 172.21.15.62:0/1300565019 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f8874036420 0x7f88740388e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f88940a0600 tx=0x7f8884008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:44.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.865+0000 7f888b7fe640 1 -- 172.21.15.62:0/1300565019 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f887c0076a0 con 0x7f88940a2900 2024-04-19T01:44:44.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.865+0000 7f888b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:44.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:44.865+0000 7f888b7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:44:45.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.093+0000 7f8899324640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:44:45.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.093+0000 7f8899324640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:44:45.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.093+0000 7f8899324640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:44:45.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.093+0000 7f8899324640 1 -- 172.21.15.62:0/1300565019 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8894127ec0 con 0x7f88940a2900 2024-04-19T01:44:45.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.093+0000 7f888b7fe640 1 -- 172.21.15.62:0/1300565019 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f887c012920 con 0x7f88940a2900 2024-04-19T01:44:45.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.093+0000 7f888b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:44:45.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.093+0000 7f888b7fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:44:45.090 INFO:teuthology.orchestra.run.smithi062.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","modified":"2024-04-19T01:41:38.497795Z","created":"2024-04-19T01:41:38.497795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:44:45.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.097+0000 7f8899324640 1 -- 172.21.15.62:0/1300565019 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f8874036420 msgr2=0x7f88740388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:45.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.097+0000 7f8899324640 1 --2- 172.21.15.62:0/1300565019 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f8874036420 0x7f88740388e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f88940a0600 tx=0x7f8884008040 comp rx=0 tx=0).stop 2024-04-19T01:44:45.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.097+0000 7f8899324640 10 monclient: shutdown 2024-04-19T01:44:45.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.097+0000 7f8899324640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:45.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.097+0000 7f8899324640 1 -- 172.21.15.62:0/1300565019 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88940a2900 msgr2=0x7f88941cf160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:45.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.097+0000 7f8899324640 1 --2- 172.21.15.62:0/1300565019 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88940a2900 0x7f88941cf160 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f887c00c440 tx=0x7f887c00c900 comp rx=0 tx=0).stop 2024-04-19T01:44:45.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.097+0000 7f8899324640 1 -- 172.21.15.62:0/1300565019 shutdown_connections 2024-04-19T01:44:45.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.097+0000 7f8899324640 1 --2- 172.21.15.62:0/1300565019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88940a3410 0x7f8894124070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.097+0000 7f8899324640 1 --2- 172.21.15.62:0/1300565019 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88940a2d50 0x7f889413d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.097+0000 7f8899324640 1 --2- 172.21.15.62:0/1300565019 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f8874036420 0x7f88740388e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.097+0000 7f8899324640 1 --2- 172.21.15.62:0/1300565019 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88940a2900 0x7f88941cf160 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.097+0000 7f8899324640 1 -- 172.21.15.62:0/1300565019 >> 172.21.15.62:0/1300565019 conn(0x7f88940aa2d0 msgr2=0x7f88940a7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:45.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.097+0000 7f8899324640 1 -- 172.21.15.62:0/1300565019 shutdown_connections 2024-04-19T01:44:45.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.097+0000 7f8899324640 1 -- 172.21.15.62:0/1300565019 wait complete. 2024-04-19T01:44:45.109 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-19T01:44:45.109 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-19T01:44:45.110 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-19T01:44:45.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.241+0000 7f97b5054640 10 monclient: get_monmap_and_config 2024-04-19T01:44:45.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.241+0000 7f97b5054640 10 monclient: build_initial_monmap 2024-04-19T01:44:45.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97b5054640 10 monclient: monmap: 2024-04-19T01:44:45.241 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:45.241 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:45.241 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:44:45.246070+0000 2024-04-19T01:44:45.241 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:44:45.246070+0000 2024-04-19T01:44:45.241 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:45.241 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:45.241 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:44:45.241 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:44:45.241 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:44:45.241 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:44:45.241 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:44:45.241 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:45.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97b5054640 1 Processor -- start 2024-04-19T01:44:45.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97b5054640 1 -- start start 2024-04-19T01:44:45.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97b5054640 10 monclient: init 2024-04-19T01:44:45.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97b5054640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:45.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97b5054640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-04-19T01:44:45.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97b5054640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97b009bf10 0x7f97b00a1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:45.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97b5054640 10 monclient(hunting): picked mon.noname-b con 0x7f97b009bf10 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:44:45.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97b5054640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f97b0072ef0 0x7f97b00a20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:45.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97b5054640 10 monclient(hunting): picked mon.noname-c con 0x7f97b0072ef0 addr v2:172.21.15.71:3300/0 2024-04-19T01:44:45.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97b5054640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f97b009c2f0 addr v1:172.21.15.71:6789/0 2024-04-19T01:44:45.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97b5054640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97b009a420 con 0x7f97b009c2f0 2024-04-19T01:44:45.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97b5054640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:45.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97b5054640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f97b009ca70 con 0x7f97b0072ef0 2024-04-19T01:44:45.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97b5054640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:45.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97b5054640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f97b0138490 con 0x7f97b009bf10 2024-04-19T01:44:45.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97b5054640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:45.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97b5054640 10 monclient(hunting): authenticate will time out at 1071.065632s 2024-04-19T01:44:45.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97aed9d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97b009bf10 0x7f97b00a1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:45.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97aed9d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97b009bf10 0x7f97b00a1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:51436/0 (socket says 172.21.15.62:51436) 2024-04-19T01:44:45.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97aed9d640 1 -- 172.21.15.62:0/1081689944 learned_addr learned my addr 172.21.15.62:0/1081689944 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:44:45.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97aed9d640 10 monclient(hunting): get_auth_request con 0x7f97b009bf10 auth_method 0 2024-04-19T01:44:45.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:45.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97aed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:45.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97aed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:45.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97aed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:45.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97a659c640 1 --2- 172.21.15.62:0/1081689944 >> v2:172.21.15.71:3300/0 conn(0x7f97b0072ef0 0x7f97b00a20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:45.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97aed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:45.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97aed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:45.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97a659c640 10 monclient(hunting): get_auth_request con 0x7f97b0072ef0 auth_method 0 2024-04-19T01:44:45.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:45.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97a659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:45.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97a659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:45.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:45.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:45.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:45.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97aed9d640 10 monclient(hunting): handle_auth_done global_id 4748 payload 274 2024-04-19T01:44:45.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97aed9d640 1 -- 172.21.15.62:0/1081689944 >> v2:172.21.15.71:3300/0 conn(0x7f97b0072ef0 msgr2=0x7f97b00a20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:44:45.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97aed9d640 1 --2- 172.21.15.62:0/1081689944 >> v2:172.21.15.71:3300/0 conn(0x7f97b0072ef0 0x7f97b00a20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97aed9d640 1 -- 172.21.15.62:0/1081689944 >> v1:172.21.15.71:6789/0 conn(0x7f97b009c2f0 legacy=0x7f97b013e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:44:45.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97aed9d640 10 monclient: _finish_hunting 0 2024-04-19T01:44:45.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97aed9d640 1 monclient: found mon.noname-b 2024-04-19T01:44:45.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97aed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:45.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97aed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:44:45.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.245+0000 7f97aed9d640 1 -- 172.21.15.62:0/1081689944 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b009a9e0 con 0x7f97b009bf10 2024-04-19T01:44:45.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.249+0000 7f97aed9d640 1 -- 172.21.15.62:0/1081689944 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97b009bf10 msgr2=0x7f97b00a1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:45.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.249+0000 7f97aed9d640 1 --2- 172.21.15.62:0/1081689944 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97b009bf10 0x7f97b00a1b70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f97a0009870 tx=0x7f97a001c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4c1bdeb643370e server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:45.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.249+0000 7f97ae59c640 1 -- 172.21.15.62:0/1081689944 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a00b0070 con 0x7f97b009bf10 2024-04-19T01:44:45.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.249+0000 7f97ae59c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:45.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.249+0000 7f97ae59c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:44:45.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.249+0000 7f97ae59c640 10 monclient: dump: 2024-04-19T01:44:45.249 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:45.249 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:45.249 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.249 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.249 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:45.249 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:45.249 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:45.249 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:45.249 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:45.249 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:45.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.249+0000 7f97ae59c640 10 monclient: _finish_auth 0 2024-04-19T01:44:45.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.249+0000 7f97ae59c640 10 monclient: _check_auth_tickets 2024-04-19T01:44:45.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.249+0000 7f97ae59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:45.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.249+0000 7f97b5054640 5 monclient: authenticate success, global_id 4748 2024-04-19T01:44:45.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.249+0000 7f97ae59c640 1 -- 172.21.15.62:0/1081689944 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97a00ab600 con 0x7f97b009bf10 2024-04-19T01:44:45.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.249+0000 7f97b5054640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:44:45.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.249+0000 7f97ae59c640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:45.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.249+0000 7f97b5054640 10 monclient: get_monmap_and_config success 2024-04-19T01:44:45.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.249+0000 7f97ae59c640 1 -- 172.21.15.62:0/1081689944 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a00ab9d0 con 0x7f97b009bf10 2024-04-19T01:44:45.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.249+0000 7f97ae59c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:45.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.249+0000 7f97b5054640 4 set_mon_vals no callback set 2024-04-19T01:44:45.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.249+0000 7f97ae59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:44:45.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.249+0000 7f97ae59c640 10 monclient: dump: 2024-04-19T01:44:45.250 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:45.250 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:45.250 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.250 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.250 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:45.251 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:45.251 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:45.251 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:45.251 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:45.251 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:45.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.249+0000 7f97b5054640 10 monclient: shutdown 2024-04-19T01:44:45.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.249+0000 7f97b5054640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:45.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.249+0000 7f97b5054640 1 -- 172.21.15.62:0/1081689944 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97b009bf10 msgr2=0x7f97b00a1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:45.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.249+0000 7f97b5054640 1 --2- 172.21.15.62:0/1081689944 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97b009bf10 0x7f97b00a1b70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f97a0009870 tx=0x7f97a001c420 comp rx=0 tx=0).stop 2024-04-19T01:44:45.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.249+0000 7f97b5054640 1 -- 172.21.15.62:0/1081689944 shutdown_connections 2024-04-19T01:44:45.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.249+0000 7f97b5054640 1 --2- 172.21.15.62:0/1081689944 >> v2:172.21.15.71:3300/0 conn(0x7f97b0072ef0 0x7f97b00a20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.249+0000 7f97b5054640 1 --2- 172.21.15.62:0/1081689944 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97b009bf10 0x7f97b00a1b70 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.249+0000 7f97b5054640 1 -- 172.21.15.62:0/1081689944 >> 172.21.15.62:0/1081689944 conn(0x7f97b00a2c60 msgr2=0x7f97b00a3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:45.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.249+0000 7f97b5054640 1 -- 172.21.15.62:0/1081689944 shutdown_connections 2024-04-19T01:44:45.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.249+0000 7f97b5054640 1 -- 172.21.15.62:0/1081689944 wait complete. 2024-04-19T01:44:45.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97b5054640 10 monclient: build_initial_monmap 2024-04-19T01:44:45.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97b5054640 10 monclient: monmap: 2024-04-19T01:44:45.252 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:45.252 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:44:45.252 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:44:45.252 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:44:45.252 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:45.252 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:45.252 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:44:45.252 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:44:45.252 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:44:45.252 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:45.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97b5054640 1 Processor -- start 2024-04-19T01:44:45.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97b5054640 1 -- start start 2024-04-19T01:44:45.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97b5054640 10 monclient: init 2024-04-19T01:44:45.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97b5054640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:45.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97b5054640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-19T01:44:45.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97b5054640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97b0072ef0 0x7f97b013e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:45.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97b5054640 10 monclient(hunting): picked mon.noname-b con 0x7f97b0072ef0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:44:45.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97b5054640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97b009bf10 0x7f97b01367c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:45.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97b5054640 10 monclient(hunting): picked mon.noname-a con 0x7f97b009bf10 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:44:45.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97b5054640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97b009c2f0 0x7f97b0136d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:45.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97b5054640 10 monclient(hunting): picked mon.noname-c con 0x7f97b009c2f0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:44:45.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97b5054640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:45.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97b5054640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f97b01427d0 con 0x7f97b009bf10 2024-04-19T01:44:45.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97b5054640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:45.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97b5054640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f97b0137c50 con 0x7f97b0072ef0 2024-04-19T01:44:45.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97b5054640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:45.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97b5054640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f97b0142950 con 0x7f97b009c2f0 2024-04-19T01:44:45.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97b5054640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:45.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97b5054640 10 monclient(hunting): authenticate will time out at 1071.073169s 2024-04-19T01:44:45.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97a659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97b009bf10 0x7f97b01367c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:45.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97a659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97b009bf10 0x7f97b01367c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54146/0 (socket says 172.21.15.62:54146) 2024-04-19T01:44:45.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97af59e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97b009c2f0 0x7f97b0136d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:45.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97a659c640 1 -- 172.21.15.62:0/156278695 learned_addr learned my addr 172.21.15.62:0/156278695 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:44:45.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97a659c640 10 monclient(hunting): get_auth_request con 0x7f97b009bf10 auth_method 0 2024-04-19T01:44:45.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:45.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97a659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:45.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97a659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:45.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97af59e640 10 monclient(hunting): get_auth_request con 0x7f97b009c2f0 auth_method 0 2024-04-19T01:44:45.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97af59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:45.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97af59e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:45.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97af59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:45.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97aed9d640 1 --2- 172.21.15.62:0/156278695 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97b0072ef0 0x7f97b013e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:45.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:45.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:45.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:45.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97af59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:45.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97af59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:45.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97af59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:45.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97aed9d640 10 monclient(hunting): get_auth_request con 0x7f97b0072ef0 auth_method 0 2024-04-19T01:44:45.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:45.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97aed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:45.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97aed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:45.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97af59e640 10 monclient(hunting): handle_auth_done global_id 4751 payload 824 2024-04-19T01:44:45.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97af59e640 1 -- 172.21.15.62:0/156278695 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97b0072ef0 msgr2=0x7f97b013e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:45.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97af59e640 1 --2- 172.21.15.62:0/156278695 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97b0072ef0 0x7f97b013e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97af59e640 1 -- 172.21.15.62:0/156278695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97b009bf10 msgr2=0x7f97b01367c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:45.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97af59e640 1 --2- 172.21.15.62:0/156278695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97b009bf10 0x7f97b01367c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97af59e640 10 monclient: _finish_hunting 0 2024-04-19T01:44:45.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97af59e640 1 monclient: found mon.noname-c 2024-04-19T01:44:45.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97af59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:45.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97af59e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:44:45.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97af59e640 1 -- 172.21.15.62:0/156278695 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b0073c10 con 0x7f97b009c2f0 2024-04-19T01:44:45.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97a659c640 1 --2- 172.21.15.62:0/156278695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97b009bf10 0x7f97b01367c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:44:45.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.253+0000 7f97aed9d640 1 --2- 172.21.15.62:0/156278695 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97b0072ef0 0x7f97b013e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T01:44:45.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97af59e640 1 -- 172.21.15.62:0/156278695 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97b009c2f0 msgr2=0x7f97b0136d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:45.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97af59e640 1 --2- 172.21.15.62:0/156278695 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97b009c2f0 0x7f97b0136d00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f97a800c780 tx=0x7f97a800cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:45.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97a7fff640 1 -- 172.21.15.62:0/156278695 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a8016020 con 0x7f97b009c2f0 2024-04-19T01:44:45.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:45.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97a7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:44:45.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97a7fff640 10 monclient: dump: 2024-04-19T01:44:45.261 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:45.262 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:45.262 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.262 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.262 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:45.262 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:45.262 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:45.262 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:45.262 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:45.262 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:45.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97a7fff640 10 monclient: _finish_auth 0 2024-04-19T01:44:45.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97a7fff640 10 monclient: _check_auth_tickets 2024-04-19T01:44:45.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:45.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97a7fff640 1 -- 172.21.15.62:0/156278695 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97a800ac60 con 0x7f97b009c2f0 2024-04-19T01:44:45.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97b5054640 5 monclient: authenticate success, global_id 4751 2024-04-19T01:44:45.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97b5054640 10 monclient: _renew_subs 2024-04-19T01:44:45.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97b5054640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:44:45.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97b5054640 1 -- 172.21.15.62:0/156278695 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97b012e240 con 0x7f97b009c2f0 2024-04-19T01:44:45.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97a7fff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:45.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97a7fff640 1 -- 172.21.15.62:0/156278695 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a8012660 con 0x7f97b009c2f0 2024-04-19T01:44:45.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:45.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97a7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:44:45.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97a7fff640 10 monclient: dump: 2024-04-19T01:44:45.263 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:45.263 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:45.263 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.263 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.263 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:45.263 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:45.263 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:45.264 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:45.264 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:45.264 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:45.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97b5054640 10 monclient: _renew_subs 2024-04-19T01:44:45.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97b5054640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:44:45.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97b5054640 1 -- 172.21.15.62:0/156278695 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f97b013f3e0 con 0x7f97b009c2f0 2024-04-19T01:44:45.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97add9b640 4 set_mon_vals no callback set 2024-04-19T01:44:45.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97b5054640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:45.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97b5054640 1 --2- 172.21.15.62:0/156278695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97b012d9a0 0x7f97b01cfff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:45.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97b5054640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:45.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97b5054640 1 -- 172.21.15.62:0/156278695 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f97b0068ab0 con 0x7f97b012d9a0 2024-04-19T01:44:45.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97aed9d640 1 --2- 172.21.15.62:0/156278695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97b012d9a0 0x7f97b01cfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:45.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97aed9d640 10 monclient: get_auth_request con 0x7f97b012d9a0 auth_method 0 2024-04-19T01:44:45.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:45.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97aed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:45.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97aed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:45.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97a7fff640 1 -- 172.21.15.62:0/156278695 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f97a8019020 con 0x7f97b009c2f0 2024-04-19T01:44:45.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97aed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:45.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97aed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:45.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97aed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:45.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97aed9d640 10 monclient(hunting): handle_auth_done global_id 14151 payload 824 2024-04-19T01:44:45.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97aed9d640 1 -- 172.21.15.62:0/156278695 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f97b01373e0 con 0x7f97b012d9a0 2024-04-19T01:44:45.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97a7fff640 1 --2- 172.21.15.62:0/156278695 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f9790036470 0x7f9790038930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:45.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97a7fff640 1 -- 172.21.15.62:0/156278695 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(115..115 src has 1..115) ==== 4738+0+0 (secure 0 0 0) 0x7f97a8049620 con 0x7f97b009c2f0 2024-04-19T01:44:45.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97a659c640 1 --2- 172.21.15.62:0/156278695 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f9790036470 0x7f9790038930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:45.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.257+0000 7f97a659c640 10 monclient: get_auth_request con 0x7f9790036470 auth_method 0 2024-04-19T01:44:45.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.261+0000 7f97aed9d640 1 -- 172.21.15.62:0/156278695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97b012d9a0 msgr2=0x7f97b01cfff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:45.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.261+0000 7f97aed9d640 1 --2- 172.21.15.62:0/156278695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97b012d9a0 0x7f97b01cfff0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f97a00ad000 tx=0x7f97a001ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:45.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.261+0000 7f97a7fff640 1 -- 172.21.15.62:0/156278695 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a00ba020 con 0x7f97b012d9a0 2024-04-19T01:44:45.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.261+0000 7f97a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:45.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.261+0000 7f97a7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:44:45.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.261+0000 7f97a7fff640 10 monclient: dump: 2024-04-19T01:44:45.266 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:45.266 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:45.266 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.266 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.266 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:45.266 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:45.266 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:45.266 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:45.266 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:45.266 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:45.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.261+0000 7f97a7fff640 1 -- 172.21.15.62:0/156278695 <== mon.0 v2:172.21.15.62:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f97a00b6420 con 0x7f97b012d9a0 2024-04-19T01:44:45.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.261+0000 7f97a7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:45.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.261+0000 7f97a7fff640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:44:45.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.261+0000 7f97a7fff640 1 -- 172.21.15.62:0/156278695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97b012d9a0 msgr2=0x7f97b01cfff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:45.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.261+0000 7f97a7fff640 1 --2- 172.21.15.62:0/156278695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97b012d9a0 0x7f97b01cfff0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f97a00ad000 tx=0x7f97a001ac60 comp rx=0 tx=0).stop 2024-04-19T01:44:45.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.261+0000 7f97a7fff640 1 -- 172.21.15.62:0/156278695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97b012d9a0 msgr2=0x7f97b01cfff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:44:45.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.261+0000 7f97a7fff640 1 --2- 172.21.15.62:0/156278695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97b012d9a0 0x7f97b01cfff0 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f97a00ad000 tx=0x7f97a001ac60 comp rx=0 tx=0).stop 2024-04-19T01:44:45.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.261+0000 7f97a659c640 1 --2- 172.21.15.62:0/156278695 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f9790036470 0x7f9790038930 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f97b013f110 tx=0x7f9798009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:45.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.265+0000 7f97b5054640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:44:45.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.265+0000 7f97b5054640 1 --2- 172.21.15.62:0/156278695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97b01d0530 0x7f97b01d0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:45.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.265+0000 7f97b5054640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:45.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.265+0000 7f97b5054640 1 -- 172.21.15.62:0/156278695 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f97b01d0e80 con 0x7f97b01d0530 2024-04-19T01:44:45.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.265+0000 7f97aed9d640 1 --2- 172.21.15.62:0/156278695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97b01d0530 0x7f97b01d0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:45.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.265+0000 7f97aed9d640 10 monclient: get_auth_request con 0x7f97b01d0530 auth_method 0 2024-04-19T01:44:45.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.265+0000 7f97aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:45.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.265+0000 7f97aed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:45.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.265+0000 7f97aed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:45.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.265+0000 7f97aed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:45.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.265+0000 7f97aed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:45.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.265+0000 7f97aed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:45.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.265+0000 7f97aed9d640 10 monclient(hunting): handle_auth_done global_id 14154 payload 824 2024-04-19T01:44:45.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.265+0000 7f97aed9d640 1 -- 172.21.15.62:0/156278695 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f97b00679d0 con 0x7f97b01d0530 2024-04-19T01:44:45.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.269+0000 7f97aed9d640 1 -- 172.21.15.62:0/156278695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97b01d0530 msgr2=0x7f97b01d0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:45.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.269+0000 7f97aed9d640 1 --2- 172.21.15.62:0/156278695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97b01d0530 0x7f97b01d0940 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f97a0009870 tx=0x7f97a00a8000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:45.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.269+0000 7f97a7fff640 1 -- 172.21.15.62:0/156278695 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a00ba070 con 0x7f97b01d0530 2024-04-19T01:44:45.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.269+0000 7f97a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:45.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.269+0000 7f97a7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:44:45.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.269+0000 7f97a7fff640 10 monclient: dump: 2024-04-19T01:44:45.270 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:45.270 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:45.270 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.270 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.270 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:45.270 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:45.270 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:45.270 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:45.270 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:45.270 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:45.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.269+0000 7f97a7fff640 1 -- 172.21.15.62:0/156278695 <== mon.0 v2:172.21.15.62:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4548 (secure 0 0 0) 0x7f97a00c1860 con 0x7f97b01d0530 2024-04-19T01:44:45.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.269+0000 7f97a7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:44:45.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.269+0000 7f97a7fff640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:44:45.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.269+0000 7f97a7fff640 1 -- 172.21.15.62:0/156278695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97b01d0530 msgr2=0x7f97b01d0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:45.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.269+0000 7f97a7fff640 1 --2- 172.21.15.62:0/156278695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97b01d0530 0x7f97b01d0940 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f97a0009870 tx=0x7f97a00a8000 comp rx=0 tx=0).stop 2024-04-19T01:44:45.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.269+0000 7f97a7fff640 1 -- 172.21.15.62:0/156278695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97b01d0530 msgr2=0x7f97b01d0940 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:44:45.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.269+0000 7f97a7fff640 1 --2- 172.21.15.62:0/156278695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97b01d0530 0x7f97b01d0940 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f97a0009870 tx=0x7f97a00a8000 comp rx=0 tx=0).stop 2024-04-19T01:44:45.271 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-19T01:44:45.271 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "a", 2024-04-19T01:44:45.271 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 0, 2024-04-19T01:44:45.271 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "leader", 2024-04-19T01:44:45.272 INFO:teuthology.orchestra.run.smithi062.stdout: "election_epoch": 28, 2024-04-19T01:44:45.272 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 28831, 2024-04-19T01:44:45.272 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum": [ 2024-04-19T01:44:45.272 INFO:teuthology.orchestra.run.smithi062.stdout: 0, 2024-04-19T01:44:45.272 INFO:teuthology.orchestra.run.smithi062.stdout: 1, 2024-04-19T01:44:45.272 INFO:teuthology.orchestra.run.smithi062.stdout: 2 2024-04-19T01:44:45.272 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:44:45.272 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_age": 3, 2024-04-19T01:44:45.272 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:44:45.272 INFO:teuthology.orchestra.run.smithi062.stdout: "required_con": "2450521708007866372", 2024-04-19T01:44:45.272 INFO:teuthology.orchestra.run.smithi062.stdout: "required_mon": [ 2024-04-19T01:44:45.272 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:44:45.272 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:44:45.272 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:44:45.272 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:44:45.273 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:44:45.273 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:44:45.273 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:44:45.273 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:44:45.273 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:44:45.273 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:44:45.273 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:44:45.273 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:44:45.273 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:44:45.273 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_mon": [ 2024-04-19T01:44:45.273 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:44:45.273 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:44:45.273 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:44:45.273 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:44:45.273 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:44:45.274 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:44:45.274 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:44:45.274 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:44:45.274 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:44:45.274 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:44:45.274 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:44:45.274 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:45.274 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.274 INFO:teuthology.orchestra.run.smithi062.stdout: "outside_quorum": [], 2024-04-19T01:44:45.274 INFO:teuthology.orchestra.run.smithi062.stdout: "extra_probe_peers": [], 2024-04-19T01:44:45.274 INFO:teuthology.orchestra.run.smithi062.stdout: "sync_provider": [], 2024-04-19T01:44:45.274 INFO:teuthology.orchestra.run.smithi062.stdout: "monmap": { 2024-04-19T01:44:45.274 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 1, 2024-04-19T01:44:45.274 INFO:teuthology.orchestra.run.smithi062.stdout: "fsid": "6466e4b6-2520-485f-95b3-9401915fbbad", 2024-04-19T01:44:45.274 INFO:teuthology.orchestra.run.smithi062.stdout: "modified": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:44:45.275 INFO:teuthology.orchestra.run.smithi062.stdout: "created": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:44:45.275 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release": 19, 2024-04-19T01:44:45.275 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release_name": "squid", 2024-04-19T01:44:45.275 INFO:teuthology.orchestra.run.smithi062.stdout: "election_strategy": 1, 2024-04-19T01:44:45.275 INFO:teuthology.orchestra.run.smithi062.stdout: "disallowed_leaders: ": "", 2024-04-19T01:44:45.275 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false, 2024-04-19T01:44:45.275 INFO:teuthology.orchestra.run.smithi062.stdout: "tiebreaker_mon": "", 2024-04-19T01:44:45.275 INFO:teuthology.orchestra.run.smithi062.stdout: "removed_ranks: ": "", 2024-04-19T01:44:45.275 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:44:45.275 INFO:teuthology.orchestra.run.smithi062.stdout: "persistent": [ 2024-04-19T01:44:45.275 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:44:45.275 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:44:45.275 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:44:45.275 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:44:45.275 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:44:45.276 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:44:45.276 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:44:45.276 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:44:45.276 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:44:45.276 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:44:45.276 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:44:45.276 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:44:45.276 INFO:teuthology.orchestra.run.smithi062.stdout: "optional": [] 2024-04-19T01:44:45.276 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.276 INFO:teuthology.orchestra.run.smithi062.stdout: "mons": [ 2024-04-19T01:44:45.276 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.276 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 0, 2024-04-19T01:44:45.276 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "a", 2024-04-19T01:44:45.276 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:44:45.276 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:44:45.277 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.277 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:44:45.277 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3300", 2024-04-19T01:44:45.277 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:45.277 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.277 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.277 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:44:45.277 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789", 2024-04-19T01:44:45.277 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:45.277 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:45.277 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:45.277 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.277 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789/0", 2024-04-19T01:44:45.277 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6789/0", 2024-04-19T01:44:45.277 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:44:45.277 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:44:45.278 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:44:45.278 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.278 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.278 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 1, 2024-04-19T01:44:45.278 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "b", 2024-04-19T01:44:45.278 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:44:45.278 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:44:45.278 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.278 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:44:45.278 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:3300", 2024-04-19T01:44:45.278 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:45.278 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.278 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.278 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:44:45.278 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789", 2024-04-19T01:44:45.279 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:45.279 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:45.279 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:45.279 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.279 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789/0", 2024-04-19T01:44:45.279 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.71:6789/0", 2024-04-19T01:44:45.279 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:44:45.279 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:44:45.279 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:44:45.279 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.279 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.279 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 2, 2024-04-19T01:44:45.279 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "c", 2024-04-19T01:44:45.279 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:44:45.279 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:44:45.280 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.280 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:44:45.280 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3301", 2024-04-19T01:44:45.280 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:45.280 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.280 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.280 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:44:45.280 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790", 2024-04-19T01:44:45.280 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:45.280 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:45.280 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:45.280 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.280 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790/0", 2024-04-19T01:44:45.280 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6790/0", 2024-04-19T01:44:45.280 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:44:45.281 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:44:45.281 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:44:45.281 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:45.281 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:45.281 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.281 INFO:teuthology.orchestra.run.smithi062.stdout: "feature_map": { 2024-04-19T01:44:45.281 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": [ 2024-04-19T01:44:45.281 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.281 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:44:45.281 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:44:45.281 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:44:45.281 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:45.281 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:44:45.281 INFO:teuthology.orchestra.run.smithi062.stdout: "client": [ 2024-04-19T01:44:45.281 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.282 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:44:45.282 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:44:45.282 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:44:45.282 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:45.282 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:45.282 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.282 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false 2024-04-19T01:44:45.282 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-19T01:44:45.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.269+0000 7f97b5054640 1 -- 172.21.15.62:0/156278695 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f9790036470 msgr2=0x7f9790038930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:45.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.269+0000 7f97b5054640 1 --2- 172.21.15.62:0/156278695 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f9790036470 0x7f9790038930 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f97b013f110 tx=0x7f9798009190 comp rx=0 tx=0).stop 2024-04-19T01:44:45.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.269+0000 7f97b5054640 10 monclient: shutdown 2024-04-19T01:44:45.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.269+0000 7f97b5054640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:45.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.269+0000 7f97b5054640 1 -- 172.21.15.62:0/156278695 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97b009c2f0 msgr2=0x7f97b0136d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:45.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.269+0000 7f97b5054640 1 --2- 172.21.15.62:0/156278695 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f97b009c2f0 0x7f97b0136d00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f97a800c780 tx=0x7f97a800cc40 comp rx=0 tx=0).stop 2024-04-19T01:44:45.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.269+0000 7f97af59e640 1 -- 172.21.15.62:0/156278695 reap_dead start 2024-04-19T01:44:45.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.269+0000 7f97af59e640 1 -- 172.21.15.62:0/156278695 reap_dead start 2024-04-19T01:44:45.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.269+0000 7f97b5054640 1 -- 172.21.15.62:0/156278695 shutdown_connections 2024-04-19T01:44:45.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.269+0000 7f97b5054640 1 -- 172.21.15.62:0/156278695 >> 172.21.15.62:0/156278695 conn(0x7f97b00a2c60 msgr2=0x7f97b009e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:45.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.269+0000 7f97b5054640 1 -- 172.21.15.62:0/156278695 shutdown_connections 2024-04-19T01:44:45.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.269+0000 7f97b5054640 1 -- 172.21.15.62:0/156278695 wait complete. 2024-04-19T01:44:45.285 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-19T01:44:45.285 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-19T01:44:45.285 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-19T01:44:45.296 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28873]: starting. 2024-04-19T01:44:45.296 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28873]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-28872 2024-04-19T01:44:45.297 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28873]: created object 0... 2024-04-19T01:44:45.297 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28873]: created object 25... 2024-04-19T01:44:45.297 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28873]: created object 49... 2024-04-19T01:44:45.297 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28873]: finishing. 2024-04-19T01:44:45.297 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28873]: shutting down. 2024-04-19T01:44:45.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f3481020640 10 monclient: get_monmap_and_config 2024-04-19T01:44:45.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f3481020640 10 monclient: build_initial_monmap 2024-04-19T01:44:45.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f3481020640 10 monclient: monmap: 2024-04-19T01:44:45.398 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:45.399 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:45.399 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:44:45.403677+0000 2024-04-19T01:44:45.399 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:44:45.403677+0000 2024-04-19T01:44:45.399 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:45.399 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:45.399 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:44:45.399 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:44:45.399 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:44:45.399 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:44:45.400 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:44:45.400 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:45.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f3481020640 1 Processor -- start 2024-04-19T01:44:45.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f3481020640 1 -- start start 2024-04-19T01:44:45.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f3481020640 10 monclient: init 2024-04-19T01:44:45.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f3481020640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:45.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f3481020640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-04-19T01:44:45.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f3481020640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f347c0a47e0 0x7f347c0a4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:45.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f3481020640 10 monclient(hunting): picked mon.noname-a con 0x7f347c0a47e0 addr v2:172.21.15.62:3300/0 2024-04-19T01:44:45.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f3481020640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f347c0a5100 addr v1:172.21.15.62:6789/0 2024-04-19T01:44:45.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f3481020640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f347c1357c0 0x7f347c13d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:45.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f3481020640 10 monclient(hunting): picked mon.noname-b con 0x7f347c1357c0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:44:45.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f3481020640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f347c09f7c0 con 0x7f347c0a5100 2024-04-19T01:44:45.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f3481020640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:45.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f3481020640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f347c09bf00 con 0x7f347c0a47e0 2024-04-19T01:44:45.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f3481020640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:45.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f3481020640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f347c09ee90 con 0x7f347c1357c0 2024-04-19T01:44:45.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f3481020640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:45.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f3481020640 10 monclient(hunting): authenticate will time out at 1071.222867s 2024-04-19T01:44:45.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f347ad9d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f347c0a47e0 0x7f347c0a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:45.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f347a59c640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f347c0a5100 0x7f347c1350b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42738/0 (socket says 172.21.15.62:42738) 2024-04-19T01:44:45.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f347b59e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f347c1357c0 0x7f347c13d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:45.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f347ad9d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f347c0a47e0 0x7f347c0a4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54176/0 (socket says 172.21.15.62:54176) 2024-04-19T01:44:45.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f347a59c640 1 -- 172.21.15.62:0/3070895037 learned_addr learned my addr 172.21.15.62:0/3070895037 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:44:45.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f347ad9d640 10 monclient(hunting): get_auth_request con 0x7f347c0a47e0 auth_method 0 2024-04-19T01:44:45.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f347ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:45.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f347ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:45.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f347ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:45.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f347b59e640 10 monclient(hunting): get_auth_request con 0x7f347c1357c0 auth_method 0 2024-04-19T01:44:45.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f347b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:45.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f347b59e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:45.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f347b59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:45.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f347ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:45.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f347ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:45.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f347ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:45.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f347b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:45.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f347b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:45.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f347b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:45.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f347ad9d640 10 monclient(hunting): handle_auth_done global_id 14157 payload 274 2024-04-19T01:44:45.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f347ad9d640 1 -- 172.21.15.62:0/3070895037 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f347c1357c0 msgr2=0x7f347c13d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:44:45.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f347ad9d640 1 --2- 172.21.15.62:0/3070895037 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f347c1357c0 0x7f347c13d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f347ad9d640 1 -- 172.21.15.62:0/3070895037 >> v1:172.21.15.62:6789/0 conn(0x7f347c0a5100 legacy=0x7f347c1350b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:44:45.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f347ad9d640 10 monclient: _finish_hunting 0 2024-04-19T01:44:45.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f347ad9d640 1 monclient: found mon.noname-a 2024-04-19T01:44:45.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f347ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:45.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f347ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:44:45.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.401+0000 7f347ad9d640 1 -- 172.21.15.62:0/3070895037 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f347c09c2d0 con 0x7f347c0a47e0 2024-04-19T01:44:45.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.405+0000 7f347ad9d640 1 -- 172.21.15.62:0/3070895037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f347c0a47e0 msgr2=0x7f347c0a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:45.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.405+0000 7f347ad9d640 1 --2- 172.21.15.62:0/3070895037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f347c0a47e0 0x7f347c0a4bc0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f346c0097f0 tx=0x7f346c01c210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=986c2595e6810740 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:45.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.405+0000 7f3479d9b640 1 -- 172.21.15.62:0/3070895037 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346c0b0070 con 0x7f347c0a47e0 2024-04-19T01:44:45.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.405+0000 7f3479d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:45.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.405+0000 7f3479d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:44:45.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.405+0000 7f3479d9b640 10 monclient: dump: 2024-04-19T01:44:45.408 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:45.409 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:45.409 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.409 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.409 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:45.409 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:45.409 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:45.409 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:45.409 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:45.409 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:45.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.405+0000 7f3479d9b640 10 monclient: _finish_auth 0 2024-04-19T01:44:45.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.405+0000 7f3479d9b640 10 monclient: _check_auth_tickets 2024-04-19T01:44:45.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.405+0000 7f3479d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:45.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.405+0000 7f3481020640 5 monclient: authenticate success, global_id 14157 2024-04-19T01:44:45.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.405+0000 7f3481020640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:44:45.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.405+0000 7f3479d9b640 1 -- 172.21.15.62:0/3070895037 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f346c0ab600 con 0x7f347c0a47e0 2024-04-19T01:44:45.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.405+0000 7f3479d9b640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:45.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.405+0000 7f3481020640 10 monclient: get_monmap_and_config success 2024-04-19T01:44:45.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.405+0000 7f3481020640 4 set_mon_vals no callback set 2024-04-19T01:44:45.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.405+0000 7f3479d9b640 1 -- 172.21.15.62:0/3070895037 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346c0ab9d0 con 0x7f347c0a47e0 2024-04-19T01:44:45.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.405+0000 7f3479d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:45.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.405+0000 7f3479d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:44:45.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.405+0000 7f3479d9b640 10 monclient: dump: 2024-04-19T01:44:45.410 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:45.410 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:45.410 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.410 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.410 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:45.410 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:45.410 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:45.411 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:45.411 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:45.411 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:45.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f3481020640 10 monclient: shutdown 2024-04-19T01:44:45.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f3481020640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:45.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f3481020640 1 -- 172.21.15.62:0/3070895037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f347c0a47e0 msgr2=0x7f347c0a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:45.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f3481020640 1 --2- 172.21.15.62:0/3070895037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f347c0a47e0 0x7f347c0a4bc0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f346c0097f0 tx=0x7f346c01c210 comp rx=0 tx=0).stop 2024-04-19T01:44:45.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f3481020640 1 -- 172.21.15.62:0/3070895037 shutdown_connections 2024-04-19T01:44:45.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f3481020640 1 --2- 172.21.15.62:0/3070895037 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f347c1357c0 0x7f347c13d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f3481020640 1 --2- 172.21.15.62:0/3070895037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f347c0a47e0 0x7f347c0a4bc0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f3481020640 1 -- 172.21.15.62:0/3070895037 >> 172.21.15.62:0/3070895037 conn(0x7f347c0a23d0 msgr2=0x7f347c0a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:45.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f3481020640 1 -- 172.21.15.62:0/3070895037 shutdown_connections 2024-04-19T01:44:45.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f3481020640 1 -- 172.21.15.62:0/3070895037 wait complete. 2024-04-19T01:44:45.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f3481020640 10 monclient: build_initial_monmap 2024-04-19T01:44:45.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f3481020640 10 monclient: monmap: 2024-04-19T01:44:45.411 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:45.412 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:44:45.412 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:44:45.412 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:44:45.412 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:45.412 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:45.412 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:44:45.412 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:44:45.412 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:44:45.412 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:45.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f3481020640 1 Processor -- start 2024-04-19T01:44:45.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f3481020640 1 -- start start 2024-04-19T01:44:45.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f3481020640 10 monclient: init 2024-04-19T01:44:45.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f3481020640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:45.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f3481020640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-19T01:44:45.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f3481020640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f347c0a47e0 0x7f347c1cf500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:45.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f3481020640 10 monclient(hunting): picked mon.noname-b con 0x7f347c0a47e0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:44:45.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f3481020640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f347c0a5100 0x7f347c1cfa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:45.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f3481020640 10 monclient(hunting): picked mon.noname-a con 0x7f347c0a5100 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:44:45.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f3481020640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f347c1357c0 0x7f347c1cff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:45.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f3481020640 10 monclient(hunting): picked mon.noname-c con 0x7f347c1357c0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:44:45.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f3481020640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:45.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f3481020640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f347c09c8e0 con 0x7f347c0a5100 2024-04-19T01:44:45.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f3481020640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:45.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f3481020640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f347c09c760 con 0x7f347c0a47e0 2024-04-19T01:44:45.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f3481020640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:45.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f3481020640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f347c140360 con 0x7f347c1357c0 2024-04-19T01:44:45.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f3481020640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:45.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f3481020640 10 monclient(hunting): authenticate will time out at 1071.229947s 2024-04-19T01:44:45.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f347a59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f347c0a5100 0x7f347c1cfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:45.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f347a59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f347c0a5100 0x7f347c1cfa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54178/0 (socket says 172.21.15.62:54178) 2024-04-19T01:44:45.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f347a59c640 1 -- 172.21.15.62:0/1653193756 learned_addr learned my addr 172.21.15.62:0/1653193756 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:44:45.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f347a59c640 10 monclient(hunting): get_auth_request con 0x7f347c0a5100 auth_method 0 2024-04-19T01:44:45.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f347a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:45.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f347a59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:45.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f347a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:45.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f347a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:45.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f347a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:45.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f347b59e640 1 --2- 172.21.15.62:0/1653193756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f347c1357c0 0x7f347c1cff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:45.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f347a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:45.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f347b59e640 10 monclient(hunting): get_auth_request con 0x7f347c1357c0 auth_method 0 2024-04-19T01:44:45.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f347b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:45.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f347b59e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:45.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f347b59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:45.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f347b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:45.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f347b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:45.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f347b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:45.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f347a59c640 10 monclient(hunting): handle_auth_done global_id 14160 payload 824 2024-04-19T01:44:45.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f347ad9d640 1 --2- 172.21.15.62:0/1653193756 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f347c0a47e0 0x7f347c1cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:45.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f347a59c640 1 -- 172.21.15.62:0/1653193756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f347c1357c0 msgr2=0x7f347c1cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:45.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f347a59c640 1 --2- 172.21.15.62:0/1653193756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f347c1357c0 0x7f347c1cff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f347a59c640 1 -- 172.21.15.62:0/1653193756 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f347c0a47e0 msgr2=0x7f347c1cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:45.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f347a59c640 1 --2- 172.21.15.62:0/1653193756 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f347c0a47e0 0x7f347c1cf500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f347a59c640 10 monclient: _finish_hunting 0 2024-04-19T01:44:45.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f347a59c640 1 monclient: found mon.noname-a 2024-04-19T01:44:45.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f347a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:45.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f347a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:44:45.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.409+0000 7f347a59c640 1 -- 172.21.15.62:0/1653193756 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f347c09eb40 con 0x7f347c0a5100 2024-04-19T01:44:45.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f347a59c640 1 -- 172.21.15.62:0/1653193756 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f347c0a5100 msgr2=0x7f347c1cfa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:45.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f347a59c640 1 --2- 172.21.15.62:0/1653193756 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f347c0a5100 0x7f347c1cfa40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f347000c470 tx=0x7f347000c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:45.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f34637fe640 1 -- 172.21.15.62:0/1653193756 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3470007070 con 0x7f347c0a5100 2024-04-19T01:44:45.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f34637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:45.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f34637fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:44:45.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f34637fe640 10 monclient: dump: 2024-04-19T01:44:45.420 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:45.421 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:45.421 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.421 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.421 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:45.421 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:45.421 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:45.421 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:45.421 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:45.421 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:45.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f34637fe640 10 monclient: _finish_auth 0 2024-04-19T01:44:45.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f34637fe640 10 monclient: _check_auth_tickets 2024-04-19T01:44:45.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f34637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:45.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f34637fe640 1 -- 172.21.15.62:0/1653193756 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3470007210 con 0x7f347c0a5100 2024-04-19T01:44:45.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f3481020640 5 monclient: authenticate success, global_id 14160 2024-04-19T01:44:45.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f3481020640 10 monclient: _renew_subs 2024-04-19T01:44:45.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f3481020640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:44:45.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f3481020640 1 -- 172.21.15.62:0/1653193756 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f347c13cf90 con 0x7f347c0a5100 2024-04-19T01:44:45.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f34637fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:45.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f34637fe640 1 -- 172.21.15.62:0/1653193756 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3470007600 con 0x7f347c0a5100 2024-04-19T01:44:45.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f34637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:45.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f34637fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:44:45.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f34637fe640 10 monclient: dump: 2024-04-19T01:44:45.422 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:45.422 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:45.422 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.423 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.423 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:45.423 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:45.423 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:45.423 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:45.423 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:45.423 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:45.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f347959a640 4 set_mon_vals no callback set 2024-04-19T01:44:45.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f3481020640 10 monclient: _renew_subs 2024-04-19T01:44:45.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f3481020640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:44:45.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f3481020640 1 -- 172.21.15.62:0/1653193756 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f347c09e460 con 0x7f347c0a5100 2024-04-19T01:44:45.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f3481020640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:45.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f3481020640 1 --2- 172.21.15.62:0/1653193756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f347c12e040 0x7f347c12e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:45.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f3481020640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:45.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f3481020640 1 -- 172.21.15.62:0/1653193756 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f347c0692b0 con 0x7f347c12e040 2024-04-19T01:44:45.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f347ad9d640 1 --2- 172.21.15.62:0/1653193756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f347c12e040 0x7f347c12e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:45.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f347ad9d640 10 monclient: get_auth_request con 0x7f347c12e040 auth_method 0 2024-04-19T01:44:45.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f347ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:45.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f347ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:45.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f347ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:45.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f34637fe640 1 -- 172.21.15.62:0/1653193756 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f347001b400 con 0x7f347c0a5100 2024-04-19T01:44:45.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f347ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:45.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f347ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:45.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f347ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:45.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f34637fe640 1 --2- 172.21.15.62:0/1653193756 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f3468036420 0x7f34680388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:45.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f347ad9d640 10 monclient(hunting): handle_auth_done global_id 4760 payload 824 2024-04-19T01:44:45.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f34637fe640 1 -- 172.21.15.62:0/1653193756 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 4738+0+0 (secure 0 0 0) 0x7f3470049200 con 0x7f347c0a5100 2024-04-19T01:44:45.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f347ad9d640 1 -- 172.21.15.62:0/1653193756 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f347c0008e0 con 0x7f347c12e040 2024-04-19T01:44:45.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f347b59e640 1 --2- 172.21.15.62:0/1653193756 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f3468036420 0x7f34680388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:45.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.413+0000 7f347b59e640 10 monclient: get_auth_request con 0x7f3468036420 auth_method 0 2024-04-19T01:44:45.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.417+0000 7f347ad9d640 1 -- 172.21.15.62:0/1653193756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f347c12e040 msgr2=0x7f347c12e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:45.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.417+0000 7f347ad9d640 1 --2- 172.21.15.62:0/1653193756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f347c12e040 0x7f347c12e420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f346c0ad000 tx=0x7f346c01aa50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:45.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.417+0000 7f34637fe640 1 -- 172.21.15.62:0/1653193756 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346c0bb020 con 0x7f347c12e040 2024-04-19T01:44:45.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.417+0000 7f34637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:45.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.417+0000 7f34637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:44:45.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.417+0000 7f34637fe640 10 monclient: dump: 2024-04-19T01:44:45.425 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:45.425 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:45.425 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.425 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.425 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:45.425 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:45.425 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:45.425 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:45.425 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:45.425 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:45.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.417+0000 7f34637fe640 1 -- 172.21.15.62:0/1653193756 <== mon.2 v2:172.21.15.62:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f346c0abe20 con 0x7f347c12e040 2024-04-19T01:44:45.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.417+0000 7f34637fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:45.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.417+0000 7f34637fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:44:45.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.417+0000 7f34637fe640 1 -- 172.21.15.62:0/1653193756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f347c12e040 msgr2=0x7f347c12e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:45.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.417+0000 7f34637fe640 1 --2- 172.21.15.62:0/1653193756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f347c12e040 0x7f347c12e420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f346c0ad000 tx=0x7f346c01aa50 comp rx=0 tx=0).stop 2024-04-19T01:44:45.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.417+0000 7f34637fe640 1 -- 172.21.15.62:0/1653193756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f347c12e040 msgr2=0x7f347c12e420 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:44:45.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.417+0000 7f34637fe640 1 --2- 172.21.15.62:0/1653193756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f347c12e040 0x7f347c12e420 secure :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f346c0ad000 tx=0x7f346c01aa50 comp rx=0 tx=0).stop 2024-04-19T01:44:45.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.417+0000 7f347b59e640 1 --2- 172.21.15.62:0/1653193756 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f3468036420 0x7f34680388e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f347c13c4b0 tx=0x7f347400b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:45.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.421+0000 7f3481020640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:44:45.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.421+0000 7f3481020640 1 --2- 172.21.15.62:0/1653193756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f347c12e960 0x7f347c070450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:45.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.421+0000 7f3481020640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:45.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.421+0000 7f3481020640 1 -- 172.21.15.62:0/1653193756 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f347c12ed70 con 0x7f347c12e960 2024-04-19T01:44:45.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.421+0000 7f347ad9d640 1 --2- 172.21.15.62:0/1653193756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f347c12e960 0x7f347c070450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:45.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.421+0000 7f347ad9d640 10 monclient: get_auth_request con 0x7f347c12e960 auth_method 0 2024-04-19T01:44:45.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.421+0000 7f347ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:45.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.421+0000 7f347ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:45.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.421+0000 7f347ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:45.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.421+0000 7f347ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:45.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.421+0000 7f347ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:45.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.421+0000 7f347ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:45.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.421+0000 7f347ad9d640 10 monclient(hunting): handle_auth_done global_id 4763 payload 824 2024-04-19T01:44:45.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.421+0000 7f347ad9d640 1 -- 172.21.15.62:0/1653193756 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f347c09cb80 con 0x7f347c12e960 2024-04-19T01:44:45.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.425+0000 7f347ad9d640 1 -- 172.21.15.62:0/1653193756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f347c12e960 msgr2=0x7f347c070450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:45.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.425+0000 7f347ad9d640 1 --2- 172.21.15.62:0/1653193756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f347c12e960 0x7f347c070450 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f346c0097f0 tx=0x7f346c018fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:45.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.425+0000 7f34637fe640 1 -- 172.21.15.62:0/1653193756 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346c0bb070 con 0x7f347c12e960 2024-04-19T01:44:45.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.425+0000 7f34637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:45.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.425+0000 7f34637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:44:45.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.425+0000 7f34637fe640 10 monclient: dump: 2024-04-19T01:44:45.428 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:45.428 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:45.428 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.428 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.428 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:45.428 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:45.429 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:45.429 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:45.429 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:45.429 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:45.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.425+0000 7f34637fe640 1 -- 172.21.15.62:0/1653193756 <== mon.2 v2:172.21.15.62:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4717 (secure 0 0 0) 0x7f346c0b6470 con 0x7f347c12e960 2024-04-19T01:44:45.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.425+0000 7f34637fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:44:45.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.425+0000 7f34637fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:44:45.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.425+0000 7f34637fe640 1 -- 172.21.15.62:0/1653193756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f347c12e960 msgr2=0x7f347c070450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:45.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.425+0000 7f34637fe640 1 --2- 172.21.15.62:0/1653193756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f347c12e960 0x7f347c070450 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f346c0097f0 tx=0x7f346c018fb0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.425+0000 7f34637fe640 1 -- 172.21.15.62:0/1653193756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f347c12e960 msgr2=0x7f347c070450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:44:45.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.425+0000 7f34637fe640 1 --2- 172.21.15.62:0/1653193756 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f347c12e960 0x7f347c070450 secure :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f346c0097f0 tx=0x7f346c018fb0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.429 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-19T01:44:45.430 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "c", 2024-04-19T01:44:45.430 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 2, 2024-04-19T01:44:45.430 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "peon", 2024-04-19T01:44:45.430 INFO:teuthology.orchestra.run.smithi062.stdout: "election_epoch": 28, 2024-04-19T01:44:45.430 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 161630, 2024-04-19T01:44:45.430 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum": [ 2024-04-19T01:44:45.430 INFO:teuthology.orchestra.run.smithi062.stdout: 0, 2024-04-19T01:44:45.430 INFO:teuthology.orchestra.run.smithi062.stdout: 1, 2024-04-19T01:44:45.430 INFO:teuthology.orchestra.run.smithi062.stdout: 2 2024-04-19T01:44:45.430 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:44:45.430 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_age": 3, 2024-04-19T01:44:45.430 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:44:45.430 INFO:teuthology.orchestra.run.smithi062.stdout: "required_con": "2450521708007866372", 2024-04-19T01:44:45.430 INFO:teuthology.orchestra.run.smithi062.stdout: "required_mon": [ 2024-04-19T01:44:45.431 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:44:45.431 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:44:45.431 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:44:45.431 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:44:45.431 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:44:45.431 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:44:45.431 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:44:45.431 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:44:45.431 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:44:45.431 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:44:45.431 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:44:45.431 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:44:45.431 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:44:45.431 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_mon": [ 2024-04-19T01:44:45.431 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:44:45.431 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:44:45.432 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:44:45.432 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:44:45.432 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:44:45.432 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:44:45.432 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:44:45.432 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:44:45.432 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:44:45.432 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:44:45.432 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:44:45.432 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:45.432 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.432 INFO:teuthology.orchestra.run.smithi062.stdout: "outside_quorum": [], 2024-04-19T01:44:45.432 INFO:teuthology.orchestra.run.smithi062.stdout: "extra_probe_peers": [], 2024-04-19T01:44:45.432 INFO:teuthology.orchestra.run.smithi062.stdout: "sync_provider": [], 2024-04-19T01:44:45.432 INFO:teuthology.orchestra.run.smithi062.stdout: "monmap": { 2024-04-19T01:44:45.432 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 1, 2024-04-19T01:44:45.433 INFO:teuthology.orchestra.run.smithi062.stdout: "fsid": "6466e4b6-2520-485f-95b3-9401915fbbad", 2024-04-19T01:44:45.433 INFO:teuthology.orchestra.run.smithi062.stdout: "modified": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:44:45.433 INFO:teuthology.orchestra.run.smithi062.stdout: "created": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:44:45.433 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release": 19, 2024-04-19T01:44:45.433 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release_name": "squid", 2024-04-19T01:44:45.433 INFO:teuthology.orchestra.run.smithi062.stdout: "election_strategy": 1, 2024-04-19T01:44:45.433 INFO:teuthology.orchestra.run.smithi062.stdout: "disallowed_leaders: ": "", 2024-04-19T01:44:45.433 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false, 2024-04-19T01:44:45.433 INFO:teuthology.orchestra.run.smithi062.stdout: "tiebreaker_mon": "", 2024-04-19T01:44:45.433 INFO:teuthology.orchestra.run.smithi062.stdout: "removed_ranks: ": "", 2024-04-19T01:44:45.433 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:44:45.433 INFO:teuthology.orchestra.run.smithi062.stdout: "persistent": [ 2024-04-19T01:44:45.433 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:44:45.433 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:44:45.433 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:44:45.434 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:44:45.434 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:44:45.434 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:44:45.434 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:44:45.434 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:44:45.434 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:44:45.434 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:44:45.434 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:44:45.434 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:44:45.434 INFO:teuthology.orchestra.run.smithi062.stdout: "optional": [] 2024-04-19T01:44:45.434 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.434 INFO:teuthology.orchestra.run.smithi062.stdout: "mons": [ 2024-04-19T01:44:45.434 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.434 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 0, 2024-04-19T01:44:45.434 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "a", 2024-04-19T01:44:45.435 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:44:45.435 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:44:45.435 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.435 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:44:45.435 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3300", 2024-04-19T01:44:45.435 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:45.435 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.435 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.435 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:44:45.435 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789", 2024-04-19T01:44:45.435 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:45.436 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:45.436 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:45.436 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.436 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789/0", 2024-04-19T01:44:45.436 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6789/0", 2024-04-19T01:44:45.436 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:44:45.436 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:44:45.436 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:44:45.436 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.436 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.437 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 1, 2024-04-19T01:44:45.437 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "b", 2024-04-19T01:44:45.437 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:44:45.437 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:44:45.437 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.437 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:44:45.437 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:3300", 2024-04-19T01:44:45.437 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:45.437 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.437 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.437 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:44:45.437 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789", 2024-04-19T01:44:45.437 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:45.437 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:45.437 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:45.437 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.438 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789/0", 2024-04-19T01:44:45.438 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.71:6789/0", 2024-04-19T01:44:45.438 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:44:45.438 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:44:45.438 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:44:45.438 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.438 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.438 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 2, 2024-04-19T01:44:45.438 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "c", 2024-04-19T01:44:45.438 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:44:45.438 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:44:45.438 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.438 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:44:45.438 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3301", 2024-04-19T01:44:45.438 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:45.439 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.439 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.439 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:44:45.439 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790", 2024-04-19T01:44:45.439 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:45.439 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:45.439 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:45.439 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.439 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790/0", 2024-04-19T01:44:45.439 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6790/0", 2024-04-19T01:44:45.439 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:44:45.439 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:44:45.439 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:44:45.439 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:45.439 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:45.439 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.440 INFO:teuthology.orchestra.run.smithi062.stdout: "feature_map": { 2024-04-19T01:44:45.440 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": [ 2024-04-19T01:44:45.440 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.440 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:44:45.440 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:44:45.440 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:44:45.440 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:45.440 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:44:45.440 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": [ 2024-04-19T01:44:45.440 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.440 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:44:45.440 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:44:45.440 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:44:45.440 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:45.440 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:44:45.441 INFO:teuthology.orchestra.run.smithi062.stdout: "client": [ 2024-04-19T01:44:45.441 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.441 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:44:45.441 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:44:45.441 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:44:45.441 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:45.441 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:45.441 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.441 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false 2024-04-19T01:44:45.441 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-19T01:44:45.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.425+0000 7f3481020640 1 -- 172.21.15.62:0/1653193756 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f3468036420 msgr2=0x7f34680388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:45.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.425+0000 7f3481020640 1 --2- 172.21.15.62:0/1653193756 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f3468036420 0x7f34680388e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f347c13c4b0 tx=0x7f347400b040 comp rx=0 tx=0).stop 2024-04-19T01:44:45.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.425+0000 7f3481020640 10 monclient: shutdown 2024-04-19T01:44:45.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.425+0000 7f3481020640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:45.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.425+0000 7f3481020640 1 -- 172.21.15.62:0/1653193756 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f347c0a5100 msgr2=0x7f347c1cfa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:45.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.425+0000 7f3481020640 1 --2- 172.21.15.62:0/1653193756 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f347c0a5100 0x7f347c1cfa40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f347000c470 tx=0x7f347000c930 comp rx=0 tx=0).stop 2024-04-19T01:44:45.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.425+0000 7f347b59e640 1 -- 172.21.15.62:0/1653193756 reap_dead start 2024-04-19T01:44:45.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.425+0000 7f3481020640 1 -- 172.21.15.62:0/1653193756 shutdown_connections 2024-04-19T01:44:45.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.425+0000 7f3481020640 1 --2- 172.21.15.62:0/1653193756 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f347c0a5100 0x7f347c1cfa40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.425+0000 7f3481020640 1 -- 172.21.15.62:0/1653193756 >> 172.21.15.62:0/1653193756 conn(0x7f347c0a23d0 msgr2=0x7f347c06f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:45.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.425+0000 7f3481020640 1 -- 172.21.15.62:0/1653193756 shutdown_connections 2024-04-19T01:44:45.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.425+0000 7f3481020640 1 -- 172.21.15.62:0/1653193756 wait complete. 2024-04-19T01:44:45.444 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-19T01:44:45.444 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-19T01:44:45.444 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T01:44:45.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.561+0000 7fdd12a76640 10 monclient: get_monmap_and_config 2024-04-19T01:44:45.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.561+0000 7fdd12a76640 10 monclient: build_initial_monmap 2024-04-19T01:44:45.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.561+0000 7fdd12a76640 10 monclient: monmap: 2024-04-19T01:44:45.560 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:45.560 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:45.560 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:44:45.565104+0000 2024-04-19T01:44:45.560 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:44:45.565104+0000 2024-04-19T01:44:45.560 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:45.560 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:45.560 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:44:45.560 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:44:45.560 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:44:45.560 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:44:45.561 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:44:45.561 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:45.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd12a76640 1 Processor -- start 2024-04-19T01:44:45.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd12a76640 1 -- start start 2024-04-19T01:44:45.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd12a76640 10 monclient: init 2024-04-19T01:44:45.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd12a76640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:45.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd12a76640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-04-19T01:44:45.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd12a76640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdd0c137780 0x7fdd0c137b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:45.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd12a76640 10 monclient(hunting): picked mon.noname-a con 0x7fdd0c137780 addr v2:172.21.15.62:3300/0 2024-04-19T01:44:45.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd12a76640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fdd0c1380a0 0x7fdd0c0a45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:45.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd12a76640 10 monclient(hunting): picked mon.noname-c con 0x7fdd0c1380a0 addr v2:172.21.15.71:3300/0 2024-04-19T01:44:45.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd12a76640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdd0c0a60b0 0x7fdd0c0a4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:45.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd12a76640 10 monclient(hunting): picked mon.noname-b con 0x7fdd0c0a60b0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:44:45.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd12a76640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:45.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd12a76640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fdd0c09bd20 con 0x7fdd0c137780 2024-04-19T01:44:45.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd12a76640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:45.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd12a76640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fdd0c09c2e0 con 0x7fdd0c1380a0 2024-04-19T01:44:45.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd12a76640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:45.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd12a76640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fdd0c09c000 con 0x7fdd0c0a60b0 2024-04-19T01:44:45.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd12a76640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:45.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd12a76640 10 monclient(hunting): authenticate will time out at 1071.385111s 2024-04-19T01:44:45.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd10812640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdd0c137780 0x7fdd0c137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:45.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd11013640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdd0c0a60b0 0x7fdd0c0a4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:45.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd10812640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdd0c137780 0x7fdd0c137b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54186/0 (socket says 172.21.15.62:54186) 2024-04-19T01:44:45.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd10812640 1 -- 172.21.15.62:0/2833301939 learned_addr learned my addr 172.21.15.62:0/2833301939 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:44:45.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd10812640 10 monclient(hunting): get_auth_request con 0x7fdd0c137780 auth_method 0 2024-04-19T01:44:45.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd10812640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:45.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd10812640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:45.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd10812640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:45.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd11013640 10 monclient(hunting): get_auth_request con 0x7fdd0c0a60b0 auth_method 0 2024-04-19T01:44:45.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd11013640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:45.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd11013640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:45.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd11013640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:45.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd10812640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:45.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd10812640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:45.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd10812640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:45.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd11013640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:45.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd11013640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:45.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd11013640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:45.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd0bfff640 1 --2- 172.21.15.62:0/2833301939 >> v2:172.21.15.71:3300/0 conn(0x7fdd0c1380a0 0x7fdd0c0a45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:45.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd0bfff640 10 monclient(hunting): get_auth_request con 0x7fdd0c1380a0 auth_method 0 2024-04-19T01:44:45.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:45.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd0bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:45.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:45.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd11013640 10 monclient(hunting): handle_auth_done global_id 4766 payload 274 2024-04-19T01:44:45.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd11013640 1 -- 172.21.15.62:0/2833301939 >> v2:172.21.15.71:3300/0 conn(0x7fdd0c1380a0 msgr2=0x7fdd0c0a45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:44:45.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd11013640 1 --2- 172.21.15.62:0/2833301939 >> v2:172.21.15.71:3300/0 conn(0x7fdd0c1380a0 0x7fdd0c0a45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd11013640 1 -- 172.21.15.62:0/2833301939 >> v2:172.21.15.62:3300/0 conn(0x7fdd0c137780 msgr2=0x7fdd0c137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:44:45.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd11013640 1 --2- 172.21.15.62:0/2833301939 >> v2:172.21.15.62:3300/0 conn(0x7fdd0c137780 0x7fdd0c137b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd11013640 10 monclient: _finish_hunting 0 2024-04-19T01:44:45.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd11013640 1 monclient: found mon.noname-b 2024-04-19T01:44:45.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd11013640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:45.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd11013640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:44:45.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd11013640 1 -- 172.21.15.62:0/2833301939 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd0c073210 con 0x7fdd0c0a60b0 2024-04-19T01:44:45.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.565+0000 7fdd10812640 1 --2- 172.21.15.62:0/2833301939 >> v2:172.21.15.62:3300/0 conn(0x7fdd0c137780 0x7fdd0c137b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:44:45.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.569+0000 7fdd11013640 1 -- 172.21.15.62:0/2833301939 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdd0c0a60b0 msgr2=0x7fdd0c0a4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:45.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.569+0000 7fdd11013640 1 --2- 172.21.15.62:0/2833301939 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdd0c0a60b0 0x7fdd0c0a4b10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c0a74f0 tx=0x7fdd0401c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f32354215c68981 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:45.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.569+0000 7fdd0b7fe640 1 -- 172.21.15.62:0/2833301939 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd040b2070 con 0x7fdd0c0a60b0 2024-04-19T01:44:45.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.569+0000 7fdd0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:45.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.569+0000 7fdd0b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:44:45.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.569+0000 7fdd0b7fe640 10 monclient: dump: 2024-04-19T01:44:45.569 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:45.569 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:45.569 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.569 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.569 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:45.569 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:45.569 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:45.569 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:45.569 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:45.569 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:45.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.569+0000 7fdd0b7fe640 10 monclient: _finish_auth 0 2024-04-19T01:44:45.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.569+0000 7fdd0b7fe640 10 monclient: _check_auth_tickets 2024-04-19T01:44:45.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.569+0000 7fdd0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:45.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.569+0000 7fdd12a76640 5 monclient: authenticate success, global_id 4766 2024-04-19T01:44:45.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.569+0000 7fdd0b7fe640 1 -- 172.21.15.62:0/2833301939 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd040a5780 con 0x7fdd0c0a60b0 2024-04-19T01:44:45.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.569+0000 7fdd12a76640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:44:45.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.569+0000 7fdd0b7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:45.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.569+0000 7fdd12a76640 10 monclient: get_monmap_and_config success 2024-04-19T01:44:45.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.569+0000 7fdd12a76640 4 set_mon_vals no callback set 2024-04-19T01:44:45.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.569+0000 7fdd0b7fe640 1 -- 172.21.15.62:0/2833301939 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd040a5b50 con 0x7fdd0c0a60b0 2024-04-19T01:44:45.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.569+0000 7fdd0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:45.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.569+0000 7fdd0b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:44:45.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.569+0000 7fdd0b7fe640 10 monclient: dump: 2024-04-19T01:44:45.570 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:45.570 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:45.570 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.570 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.570 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:45.571 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:45.571 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:45.571 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:45.571 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:45.571 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:45.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.569+0000 7fdd12a76640 10 monclient: shutdown 2024-04-19T01:44:45.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.569+0000 7fdd12a76640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:45.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.569+0000 7fdd12a76640 1 -- 172.21.15.62:0/2833301939 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdd0c0a60b0 msgr2=0x7fdd0c0a4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:45.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.569+0000 7fdd12a76640 1 --2- 172.21.15.62:0/2833301939 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdd0c0a60b0 0x7fdd0c0a4b10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c0a74f0 tx=0x7fdd0401c7f0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.569+0000 7fdd12a76640 1 -- 172.21.15.62:0/2833301939 shutdown_connections 2024-04-19T01:44:45.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.569+0000 7fdd12a76640 1 --2- 172.21.15.62:0/2833301939 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdd0c0a60b0 0x7fdd0c0a4b10 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.569+0000 7fdd12a76640 1 --2- 172.21.15.62:0/2833301939 >> v2:172.21.15.71:3300/0 conn(0x7fdd0c1380a0 0x7fdd0c0a45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.569+0000 7fdd12a76640 1 --2- 172.21.15.62:0/2833301939 >> v2:172.21.15.62:3300/0 conn(0x7fdd0c137780 0x7fdd0c137b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.569+0000 7fdd12a76640 1 -- 172.21.15.62:0/2833301939 >> 172.21.15.62:0/2833301939 conn(0x7fdd0c0aa1d0 msgr2=0x7fdd0c0a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:45.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.569+0000 7fdd12a76640 1 -- 172.21.15.62:0/2833301939 shutdown_connections 2024-04-19T01:44:45.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.569+0000 7fdd12a76640 1 -- 172.21.15.62:0/2833301939 wait complete. 2024-04-19T01:44:45.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.569+0000 7fdd12a76640 10 monclient: build_initial_monmap 2024-04-19T01:44:45.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.569+0000 7fdd12a76640 10 monclient: monmap: 2024-04-19T01:44:45.572 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:45.572 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:44:45.572 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:44:45.572 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:44:45.572 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:45.572 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:45.572 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:44:45.572 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:44:45.572 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:44:45.572 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:45.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd12a76640 1 Processor -- start 2024-04-19T01:44:45.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd12a76640 1 -- start start 2024-04-19T01:44:45.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd12a76640 10 monclient: init 2024-04-19T01:44:45.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd12a76640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:45.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd12a76640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-19T01:44:45.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd12a76640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd0c0a60b0 0x7fdd0c1cf400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:45.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd12a76640 10 monclient(hunting): picked mon.noname-b con 0x7fdd0c0a60b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:44:45.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd12a76640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd0c137780 0x7fdd0c136450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:45.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd12a76640 10 monclient(hunting): picked mon.noname-a con 0x7fdd0c137780 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:44:45.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd12a76640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdd0c1380a0 0x7fdd0c0a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:45.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd12a76640 10 monclient(hunting): picked mon.noname-c con 0x7fdd0c1380a0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:44:45.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd12a76640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:45.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd12a76640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0c141160 con 0x7fdd0c137780 2024-04-19T01:44:45.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd12a76640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:45.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd12a76640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0c0a5460 con 0x7fdd0c0a60b0 2024-04-19T01:44:45.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd12a76640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:45.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd12a76640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fdd0c1412e0 con 0x7fdd0c1380a0 2024-04-19T01:44:45.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd12a76640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:45.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd12a76640 10 monclient(hunting): authenticate will time out at 1071.392741s 2024-04-19T01:44:45.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd0bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd0c137780 0x7fdd0c136450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:45.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd11013640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdd0c1380a0 0x7fdd0c0a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:45.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd11013640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdd0c1380a0 0x7fdd0c0a0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:51488/0 (socket says 172.21.15.62:51488) 2024-04-19T01:44:45.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd11013640 1 -- 172.21.15.62:0/2587009400 learned_addr learned my addr 172.21.15.62:0/2587009400 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:44:45.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd11013640 10 monclient(hunting): get_auth_request con 0x7fdd0c1380a0 auth_method 0 2024-04-19T01:44:45.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd11013640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:45.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd11013640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:45.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd11013640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:45.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd0bfff640 10 monclient(hunting): get_auth_request con 0x7fdd0c137780 auth_method 0 2024-04-19T01:44:45.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:45.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd0bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:45.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:45.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd11013640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:45.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd11013640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:45.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd11013640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:45.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:45.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:45.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:45.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd10812640 1 --2- 172.21.15.62:0/2587009400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd0c0a60b0 0x7fdd0c1cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:45.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd11013640 10 monclient(hunting): handle_auth_done global_id 4769 payload 824 2024-04-19T01:44:45.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd11013640 1 -- 172.21.15.62:0/2587009400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd0c0a60b0 msgr2=0x7fdd0c1cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:45.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd11013640 1 --2- 172.21.15.62:0/2587009400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd0c0a60b0 0x7fdd0c1cf400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd11013640 1 -- 172.21.15.62:0/2587009400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd0c137780 msgr2=0x7fdd0c136450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:45.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd11013640 1 --2- 172.21.15.62:0/2587009400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd0c137780 0x7fdd0c136450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd11013640 10 monclient: _finish_hunting 0 2024-04-19T01:44:45.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd11013640 1 monclient: found mon.noname-c 2024-04-19T01:44:45.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd11013640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:45.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd11013640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:44:45.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd11013640 1 -- 172.21.15.62:0/2587009400 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd0c13ddd0 con 0x7fdd0c1380a0 2024-04-19T01:44:45.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd11013640 1 -- 172.21.15.62:0/2587009400 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdd0c1380a0 msgr2=0x7fdd0c0a0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:45.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd11013640 1 --2- 172.21.15.62:0/2587009400 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdd0c1380a0 0x7fdd0c0a0850 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fdd0401aea0 tx=0x7fdd04019590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:45.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd097fa640 1 -- 172.21.15.62:0/2587009400 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd040b2050 con 0x7fdd0c1380a0 2024-04-19T01:44:45.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:45.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd097fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:44:45.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd097fa640 10 monclient: dump: 2024-04-19T01:44:45.581 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:45.581 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:45.581 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.582 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.582 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:45.582 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:45.582 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:45.582 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:45.582 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:45.582 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:45.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd097fa640 10 monclient: _finish_auth 0 2024-04-19T01:44:45.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd097fa640 10 monclient: _check_auth_tickets 2024-04-19T01:44:45.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:45.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd097fa640 1 -- 172.21.15.62:0/2587009400 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd040be070 con 0x7fdd0c1380a0 2024-04-19T01:44:45.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd097fa640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:45.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd097fa640 1 -- 172.21.15.62:0/2587009400 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd040addb0 con 0x7fdd0c1380a0 2024-04-19T01:44:45.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:45.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd097fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:44:45.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd097fa640 10 monclient: dump: 2024-04-19T01:44:45.583 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:45.583 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:45.583 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.583 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.583 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:45.583 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:45.583 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:45.583 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:45.583 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:45.584 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:45.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd12a76640 5 monclient: authenticate success, global_id 4769 2024-04-19T01:44:45.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd12a76640 10 monclient: _renew_subs 2024-04-19T01:44:45.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd0affd640 4 set_mon_vals no callback set 2024-04-19T01:44:45.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd12a76640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:44:45.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd12a76640 1 -- 172.21.15.62:0/2587009400 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd0c1d0280 con 0x7fdd0c1380a0 2024-04-19T01:44:45.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd12a76640 10 monclient: _renew_subs 2024-04-19T01:44:45.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd12a76640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:44:45.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.573+0000 7fdd12a76640 1 -- 172.21.15.62:0/2587009400 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd0c121df0 con 0x7fdd0c1380a0 2024-04-19T01:44:45.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.577+0000 7fdd12a76640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:45.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.577+0000 7fdd12a76640 1 --2- 172.21.15.62:0/2587009400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd0c136990 0x7fdd0c0a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:45.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.577+0000 7fdd12a76640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:45.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.577+0000 7fdd12a76640 1 -- 172.21.15.62:0/2587009400 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0c0a3490 con 0x7fdd0c136990 2024-04-19T01:44:45.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.577+0000 7fdd097fa640 1 -- 172.21.15.62:0/2587009400 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fdd040b9450 con 0x7fdd0c1380a0 2024-04-19T01:44:45.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.577+0000 7fdd10812640 1 --2- 172.21.15.62:0/2587009400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd0c136990 0x7fdd0c0a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:45.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.577+0000 7fdd097fa640 1 --2- 172.21.15.62:0/2587009400 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fdcf0036470 0x7fdcf0038930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:45.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.577+0000 7fdd097fa640 1 -- 172.21.15.62:0/2587009400 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(115..115 src has 1..115) ==== 4738+0+0 (secure 0 0 0) 0x7fdd040e3ef0 con 0x7fdd0c1380a0 2024-04-19T01:44:45.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.577+0000 7fdd10812640 10 monclient: get_auth_request con 0x7fdd0c136990 auth_method 0 2024-04-19T01:44:45.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.577+0000 7fdd10812640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:45.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.577+0000 7fdd10812640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:45.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.577+0000 7fdd10812640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:45.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.577+0000 7fdd10812640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:45.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.577+0000 7fdd10812640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:45.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.577+0000 7fdd10812640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:45.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.577+0000 7fdd0bfff640 1 --2- 172.21.15.62:0/2587009400 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fdcf0036470 0x7fdcf0038930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:45.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.577+0000 7fdd0bfff640 10 monclient: get_auth_request con 0x7fdcf0036470 auth_method 0 2024-04-19T01:44:45.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.577+0000 7fdd10812640 10 monclient(hunting): handle_auth_done global_id 4720 payload 824 2024-04-19T01:44:45.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.577+0000 7fdd10812640 1 -- 172.21.15.62:0/2587009400 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdd0c068ab0 con 0x7fdd0c136990 2024-04-19T01:44:45.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.581+0000 7fdd10812640 1 -- 172.21.15.62:0/2587009400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd0c136990 msgr2=0x7fdd0c0a0d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:45.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.581+0000 7fdd10812640 1 --2- 172.21.15.62:0/2587009400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd0c136990 0x7fdd0c0a0d90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdd000107c0 tx=0x7fdd00010bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:45.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.581+0000 7fdd0bfff640 1 --2- 172.21.15.62:0/2587009400 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fdcf0036470 0x7fdcf0038930 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c0a20f0 tx=0x7fdcf8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:45.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.581+0000 7fdd097fa640 1 -- 172.21.15.62:0/2587009400 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd0000f040 con 0x7fdd0c136990 2024-04-19T01:44:45.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.581+0000 7fdd097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:45.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.581+0000 7fdd097fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:44:45.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.581+0000 7fdd097fa640 10 monclient: dump: 2024-04-19T01:44:45.586 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:45.586 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:45.586 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.586 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.586 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:45.586 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:45.586 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:45.586 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:45.586 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:45.587 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:45.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.581+0000 7fdd097fa640 1 -- 172.21.15.62:0/2587009400 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdd0001c400 con 0x7fdd0c136990 2024-04-19T01:44:45.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.581+0000 7fdd097fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:45.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.581+0000 7fdd097fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:44:45.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.581+0000 7fdd097fa640 1 -- 172.21.15.62:0/2587009400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd0c136990 msgr2=0x7fdd0c0a0d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:45.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.581+0000 7fdd097fa640 1 --2- 172.21.15.62:0/2587009400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd0c136990 0x7fdd0c0a0d90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdd000107c0 tx=0x7fdd00010bc0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.581+0000 7fdd097fa640 1 -- 172.21.15.62:0/2587009400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd0c136990 msgr2=0x7fdd0c0a0d90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:44:45.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.581+0000 7fdd097fa640 1 --2- 172.21.15.62:0/2587009400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd0c136990 0x7fdd0c0a0d90 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.585+0000 7fdd12a76640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:44:45.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.585+0000 7fdd12a76640 1 --2- 172.21.15.62:0/2587009400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd0c0a12d0 0x7fdd0c0a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:45.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.585+0000 7fdd12a76640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:45.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.585+0000 7fdd12a76640 1 -- 172.21.15.62:0/2587009400 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0c0a1bf0 con 0x7fdd0c0a12d0 2024-04-19T01:44:45.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.589+0000 7fdd10812640 1 --2- 172.21.15.62:0/2587009400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd0c0a12d0 0x7fdd0c0a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:45.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.589+0000 7fdd10812640 10 monclient: get_auth_request con 0x7fdd0c0a12d0 auth_method 0 2024-04-19T01:44:45.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.589+0000 7fdd10812640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:45.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.589+0000 7fdd10812640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:45.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.589+0000 7fdd10812640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:45.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.589+0000 7fdd10812640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:45.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.589+0000 7fdd10812640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:45.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.589+0000 7fdd10812640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:45.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.589+0000 7fdd10812640 10 monclient(hunting): handle_auth_done global_id 4723 payload 824 2024-04-19T01:44:45.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.589+0000 7fdd10812640 1 -- 172.21.15.62:0/2587009400 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdd0c068710 con 0x7fdd0c0a12d0 2024-04-19T01:44:45.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.589+0000 7fdd10812640 1 -- 172.21.15.62:0/2587009400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd0c0a12d0 msgr2=0x7fdd0c0a16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:45.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.589+0000 7fdd10812640 1 --2- 172.21.15.62:0/2587009400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd0c0a12d0 0x7fdd0c0a16b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fdd0001b000 tx=0x7fdd000046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:45.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.589+0000 7fdd097fa640 1 -- 172.21.15.62:0/2587009400 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd00004220 con 0x7fdd0c0a12d0 2024-04-19T01:44:45.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.589+0000 7fdd097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:45.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.589+0000 7fdd097fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:44:45.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.589+0000 7fdd097fa640 10 monclient: dump: 2024-04-19T01:44:45.591 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:45.592 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:45.592 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.592 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.592 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:45.592 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:45.592 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:45.592 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:45.592 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:45.592 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:45.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.589+0000 7fdd097fa640 1 -- 172.21.15.62:0/2587009400 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7fdd00004220 con 0x7fdd0c0a12d0 2024-04-19T01:44:45.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.589+0000 7fdd097fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:44:45.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.589+0000 7fdd097fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:44:45.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.589+0000 7fdd097fa640 1 -- 172.21.15.62:0/2587009400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd0c0a12d0 msgr2=0x7fdd0c0a16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:45.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.593+0000 7fdd097fa640 1 --2- 172.21.15.62:0/2587009400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd0c0a12d0 0x7fdd0c0a16b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fdd0001b000 tx=0x7fdd000046d0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.593+0000 7fdd097fa640 1 -- 172.21.15.62:0/2587009400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd0c0a12d0 msgr2=0x7fdd0c0a16b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:44:45.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.593+0000 7fdd097fa640 1 --2- 172.21.15.62:0/2587009400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd0c0a12d0 0x7fdd0c0a16b0 secure :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fdd0001b000 tx=0x7fdd000046d0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.593 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-19T01:44:45.593 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "b", 2024-04-19T01:44:45.593 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 1, 2024-04-19T01:44:45.593 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "peon", 2024-04-19T01:44:45.593 INFO:teuthology.orchestra.run.smithi062.stdout: "election_epoch": 28, 2024-04-19T01:44:45.593 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 161759, 2024-04-19T01:44:45.593 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum": [ 2024-04-19T01:44:45.593 INFO:teuthology.orchestra.run.smithi062.stdout: 0, 2024-04-19T01:44:45.593 INFO:teuthology.orchestra.run.smithi062.stdout: 1, 2024-04-19T01:44:45.593 INFO:teuthology.orchestra.run.smithi062.stdout: 2 2024-04-19T01:44:45.593 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:44:45.594 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_age": 3, 2024-04-19T01:44:45.594 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:44:45.594 INFO:teuthology.orchestra.run.smithi062.stdout: "required_con": "2450521708007866372", 2024-04-19T01:44:45.594 INFO:teuthology.orchestra.run.smithi062.stdout: "required_mon": [ 2024-04-19T01:44:45.594 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:44:45.594 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:44:45.594 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:44:45.594 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:44:45.594 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:44:45.594 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:44:45.594 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:44:45.594 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:44:45.594 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:44:45.594 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:44:45.595 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:44:45.595 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:44:45.595 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:44:45.595 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_mon": [ 2024-04-19T01:44:45.595 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:44:45.595 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:44:45.595 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:44:45.595 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:44:45.595 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:44:45.595 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:44:45.595 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:44:45.595 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:44:45.595 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:44:45.596 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:44:45.596 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:44:45.596 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:45.596 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.596 INFO:teuthology.orchestra.run.smithi062.stdout: "outside_quorum": [], 2024-04-19T01:44:45.596 INFO:teuthology.orchestra.run.smithi062.stdout: "extra_probe_peers": [], 2024-04-19T01:44:45.596 INFO:teuthology.orchestra.run.smithi062.stdout: "sync_provider": [], 2024-04-19T01:44:45.596 INFO:teuthology.orchestra.run.smithi062.stdout: "monmap": { 2024-04-19T01:44:45.596 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 1, 2024-04-19T01:44:45.596 INFO:teuthology.orchestra.run.smithi062.stdout: "fsid": "6466e4b6-2520-485f-95b3-9401915fbbad", 2024-04-19T01:44:45.596 INFO:teuthology.orchestra.run.smithi062.stdout: "modified": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:44:45.596 INFO:teuthology.orchestra.run.smithi062.stdout: "created": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:44:45.596 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release": 19, 2024-04-19T01:44:45.596 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release_name": "squid", 2024-04-19T01:44:45.596 INFO:teuthology.orchestra.run.smithi062.stdout: "election_strategy": 1, 2024-04-19T01:44:45.597 INFO:teuthology.orchestra.run.smithi062.stdout: "disallowed_leaders: ": "", 2024-04-19T01:44:45.597 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false, 2024-04-19T01:44:45.597 INFO:teuthology.orchestra.run.smithi062.stdout: "tiebreaker_mon": "", 2024-04-19T01:44:45.597 INFO:teuthology.orchestra.run.smithi062.stdout: "removed_ranks: ": "", 2024-04-19T01:44:45.597 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:44:45.597 INFO:teuthology.orchestra.run.smithi062.stdout: "persistent": [ 2024-04-19T01:44:45.597 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:44:45.597 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:44:45.597 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:44:45.597 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:44:45.597 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:44:45.597 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:44:45.597 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:44:45.597 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:44:45.597 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:44:45.597 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:44:45.598 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:44:45.598 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:44:45.598 INFO:teuthology.orchestra.run.smithi062.stdout: "optional": [] 2024-04-19T01:44:45.598 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.598 INFO:teuthology.orchestra.run.smithi062.stdout: "mons": [ 2024-04-19T01:44:45.598 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.598 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 0, 2024-04-19T01:44:45.598 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "a", 2024-04-19T01:44:45.598 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:44:45.598 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:44:45.598 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.599 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:44:45.599 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3300", 2024-04-19T01:44:45.599 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:45.599 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.599 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.599 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:44:45.599 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789", 2024-04-19T01:44:45.599 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:45.599 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:45.599 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:45.599 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.599 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789/0", 2024-04-19T01:44:45.599 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6789/0", 2024-04-19T01:44:45.599 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:44:45.599 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:44:45.599 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:44:45.600 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.600 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.600 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 1, 2024-04-19T01:44:45.600 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "b", 2024-04-19T01:44:45.600 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:44:45.600 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:44:45.600 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.600 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:44:45.600 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:3300", 2024-04-19T01:44:45.600 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:45.600 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.600 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.600 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:44:45.600 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789", 2024-04-19T01:44:45.600 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:45.600 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:45.601 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:45.601 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.601 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789/0", 2024-04-19T01:44:45.601 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.71:6789/0", 2024-04-19T01:44:45.601 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:44:45.601 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:44:45.601 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:44:45.601 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.601 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.601 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 2, 2024-04-19T01:44:45.601 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "c", 2024-04-19T01:44:45.601 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:44:45.602 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:44:45.602 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.602 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:44:45.602 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3301", 2024-04-19T01:44:45.602 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:45.602 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.602 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.602 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:44:45.602 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790", 2024-04-19T01:44:45.602 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:45.602 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:45.602 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:45.602 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.602 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790/0", 2024-04-19T01:44:45.602 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6790/0", 2024-04-19T01:44:45.603 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:44:45.603 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:44:45.603 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:44:45.603 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:45.603 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:45.603 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.604 INFO:teuthology.orchestra.run.smithi062.stdout: "feature_map": { 2024-04-19T01:44:45.604 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": [ 2024-04-19T01:44:45.604 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.604 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:44:45.604 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:44:45.604 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:44:45.604 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:45.604 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:44:45.604 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": [ 2024-04-19T01:44:45.604 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.604 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:44:45.604 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:44:45.605 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 5 2024-04-19T01:44:45.605 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:45.605 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:44:45.605 INFO:teuthology.orchestra.run.smithi062.stdout: "client": [ 2024-04-19T01:44:45.605 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.605 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:44:45.605 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:44:45.605 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 6 2024-04-19T01:44:45.605 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:45.605 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:44:45.605 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": [ 2024-04-19T01:44:45.605 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:45.605 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:44:45.605 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:44:45.605 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:44:45.606 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:45.606 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:45.606 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:45.606 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false 2024-04-19T01:44:45.606 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-19T01:44:45.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.593+0000 7fdd12a76640 1 -- 172.21.15.62:0/2587009400 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fdcf0036470 msgr2=0x7fdcf0038930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:45.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.593+0000 7fdd12a76640 1 --2- 172.21.15.62:0/2587009400 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fdcf0036470 0x7fdcf0038930 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c0a20f0 tx=0x7fdcf8008040 comp rx=0 tx=0).stop 2024-04-19T01:44:45.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.593+0000 7fdd12a76640 10 monclient: shutdown 2024-04-19T01:44:45.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.593+0000 7fdd12a76640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:45.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.593+0000 7fdd12a76640 1 -- 172.21.15.62:0/2587009400 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdd0c1380a0 msgr2=0x7fdd0c0a0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:45.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.593+0000 7fdd12a76640 1 --2- 172.21.15.62:0/2587009400 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdd0c1380a0 0x7fdd0c0a0850 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fdd0401aea0 tx=0x7fdd04019590 comp rx=0 tx=0).stop 2024-04-19T01:44:45.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.593+0000 7fdd11013640 1 -- 172.21.15.62:0/2587009400 reap_dead start 2024-04-19T01:44:45.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.593+0000 7fdd12a76640 1 -- 172.21.15.62:0/2587009400 shutdown_connections 2024-04-19T01:44:45.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.593+0000 7fdd12a76640 1 --2- 172.21.15.62:0/2587009400 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fdd0c1380a0 0x7fdd0c0a0850 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.593+0000 7fdd12a76640 1 -- 172.21.15.62:0/2587009400 >> 172.21.15.62:0/2587009400 conn(0x7fdd0c0aa1d0 msgr2=0x7fdd0c13b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:45.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.593+0000 7fdd12a76640 1 -- 172.21.15.62:0/2587009400 shutdown_connections 2024-04-19T01:44:45.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.593+0000 7fdd12a76640 1 -- 172.21.15.62:0/2587009400 wait complete. 2024-04-19T01:44:45.609 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-19T01:44:45.609 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-19T01:44:45.609 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-19T01:44:45.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.725+0000 7f997cfcc640 10 monclient: get_monmap_and_config 2024-04-19T01:44:45.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.725+0000 7f997cfcc640 10 monclient: build_initial_monmap 2024-04-19T01:44:45.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.725+0000 7f997cfcc640 10 monclient: monmap: 2024-04-19T01:44:45.724 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:45.724 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:45.724 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:44:45.729043+0000 2024-04-19T01:44:45.724 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:44:45.729043+0000 2024-04-19T01:44:45.724 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:45.724 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:45.724 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:44:45.724 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:44:45.724 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:44:45.725 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:44:45.725 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:44:45.725 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:45.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.725+0000 7f997cfcc640 1 Processor -- start 2024-04-19T01:44:45.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.725+0000 7f997cfcc640 1 -- start start 2024-04-19T01:44:45.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.725+0000 7f997cfcc640 10 monclient: init 2024-04-19T01:44:45.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f997cfcc640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:45.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f997cfcc640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-04-19T01:44:45.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f997cfcc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f99780a47e0 addr v1:172.21.15.71:6789/0 2024-04-19T01:44:45.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f997cfcc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f99780a52d0 0x7f997813a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:45.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f997cfcc640 10 monclient(hunting): picked mon.noname-a con 0x7f99780a52d0 addr v2:172.21.15.62:3300/0 2024-04-19T01:44:45.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f997cfcc640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f997813ab20 0x7f997813cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:45.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f997cfcc640 10 monclient(hunting): picked mon.noname-c con 0x7f997813ab20 addr v2:172.21.15.71:3300/0 2024-04-19T01:44:45.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f997cfcc640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f997809f7c0 con 0x7f99780a47e0 2024-04-19T01:44:45.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f997cfcc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:45.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f997cfcc640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f997809bf00 con 0x7f99780a52d0 2024-04-19T01:44:45.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f997cfcc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:45.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f997cfcc640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f997809ee90 con 0x7f997813ab20 2024-04-19T01:44:45.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f997cfcc640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:45.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f997cfcc640 10 monclient(hunting): authenticate will time out at 1071.548642s 2024-04-19T01:44:45.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f9975d9b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f99780a52d0 0x7f997813a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:45.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f9975d9b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f99780a52d0 0x7f997813a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54194/0 (socket says 172.21.15.62:54194) 2024-04-19T01:44:45.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f9975d9b640 1 -- 172.21.15.62:0/2558889103 learned_addr learned my addr 172.21.15.62:0/2558889103 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:44:45.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f9975d9b640 10 monclient(hunting): get_auth_request con 0x7f99780a52d0 auth_method 0 2024-04-19T01:44:45.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f9975d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:45.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f9975d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:45.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f9975d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:45.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f9975d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:45.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f9975d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:45.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f9975d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:45.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f9976d9d640 1 --2- 172.21.15.62:0/2558889103 >> v2:172.21.15.71:3300/0 conn(0x7f997813ab20 0x7f997813cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:45.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f9976d9d640 10 monclient(hunting): get_auth_request con 0x7f997813ab20 auth_method 0 2024-04-19T01:44:45.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f9976d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:45.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f9976d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:45.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f9976d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:45.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f9975d9b640 10 monclient(hunting): handle_auth_done global_id 14169 payload 274 2024-04-19T01:44:45.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f9975d9b640 1 -- 172.21.15.62:0/2558889103 >> v2:172.21.15.71:3300/0 conn(0x7f997813ab20 msgr2=0x7f997813cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:44:45.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f9975d9b640 1 --2- 172.21.15.62:0/2558889103 >> v2:172.21.15.71:3300/0 conn(0x7f997813ab20 0x7f997813cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f9975d9b640 1 -- 172.21.15.62:0/2558889103 >> v1:172.21.15.71:6789/0 conn(0x7f99780a47e0 legacy=0x7f99780a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:44:45.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f9975d9b640 10 monclient: _finish_hunting 0 2024-04-19T01:44:45.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f9975d9b640 1 monclient: found mon.noname-a 2024-04-19T01:44:45.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f9975d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:45.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f9975d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:44:45.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.729+0000 7f9975d9b640 1 -- 172.21.15.62:0/2558889103 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f997809c2d0 con 0x7f99780a52d0 2024-04-19T01:44:45.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f9975d9b640 1 -- 172.21.15.62:0/2558889103 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99780a52d0 msgr2=0x7f997813a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:45.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f9975d9b640 1 --2- 172.21.15.62:0/2558889103 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99780a52d0 0x7f997813a5e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f996c0098a0 tx=0x7f996c01c270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88462c9b4d1f522 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:45.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997559a640 1 -- 172.21.15.62:0/2558889103 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996c0b0070 con 0x7f99780a52d0 2024-04-19T01:44:45.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:45.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:44:45.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997559a640 10 monclient: dump: 2024-04-19T01:44:45.735 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:45.735 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:45.735 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.735 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.735 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:45.735 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:45.735 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:45.736 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:45.736 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:45.736 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:45.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997559a640 10 monclient: _finish_auth 0 2024-04-19T01:44:45.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997559a640 10 monclient: _check_auth_tickets 2024-04-19T01:44:45.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:45.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997559a640 1 -- 172.21.15.62:0/2558889103 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f996c0ab600 con 0x7f99780a52d0 2024-04-19T01:44:45.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 5 monclient: authenticate success, global_id 14169 2024-04-19T01:44:45.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:44:45.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997559a640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:45.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 10 monclient: get_monmap_and_config success 2024-04-19T01:44:45.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 4 set_mon_vals no callback set 2024-04-19T01:44:45.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997559a640 1 -- 172.21.15.62:0/2558889103 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996c0ab9d0 con 0x7f99780a52d0 2024-04-19T01:44:45.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:45.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:44:45.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997559a640 10 monclient: dump: 2024-04-19T01:44:45.737 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:45.737 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:45.737 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.737 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.737 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:45.737 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:45.737 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:45.737 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:45.737 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:45.737 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:45.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 10 monclient: shutdown 2024-04-19T01:44:45.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:45.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 1 -- 172.21.15.62:0/2558889103 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99780a52d0 msgr2=0x7f997813a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:45.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 1 --2- 172.21.15.62:0/2558889103 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99780a52d0 0x7f997813a5e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f996c0098a0 tx=0x7f996c01c270 comp rx=0 tx=0).stop 2024-04-19T01:44:45.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 1 -- 172.21.15.62:0/2558889103 shutdown_connections 2024-04-19T01:44:45.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 1 --2- 172.21.15.62:0/2558889103 >> v2:172.21.15.71:3300/0 conn(0x7f997813ab20 0x7f997813cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 1 --2- 172.21.15.62:0/2558889103 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99780a52d0 0x7f997813a5e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 1 -- 172.21.15.62:0/2558889103 >> 172.21.15.62:0/2558889103 conn(0x7f99780a23d0 msgr2=0x7f99780a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:45.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 1 -- 172.21.15.62:0/2558889103 shutdown_connections 2024-04-19T01:44:45.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 1 -- 172.21.15.62:0/2558889103 wait complete. 2024-04-19T01:44:45.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 10 monclient: build_initial_monmap 2024-04-19T01:44:45.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 10 monclient: monmap: 2024-04-19T01:44:45.738 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:45.738 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:44:45.738 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:44:45.738 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:44:45.738 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:45.738 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:45.739 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:44:45.739 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:44:45.739 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:44:45.739 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:45.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 1 Processor -- start 2024-04-19T01:44:45.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 1 -- start start 2024-04-19T01:44:45.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 10 monclient: init 2024-04-19T01:44:45.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:45.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-19T01:44:45.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99780a47e0 0x7f99781cd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:45.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 10 monclient(hunting): picked mon.noname-b con 0x7f99780a47e0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:44:45.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99780a52d0 0x7f99781351c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:45.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 10 monclient(hunting): picked mon.noname-a con 0x7f99780a52d0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:44:45.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f997813ab20 0x7f997811bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:45.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 10 monclient(hunting): picked mon.noname-c con 0x7f997813ab20 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:44:45.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:45.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f997813fc70 con 0x7f99780a52d0 2024-04-19T01:44:45.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:45.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f997809c970 con 0x7f99780a47e0 2024-04-19T01:44:45.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:45.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f997813fdf0 con 0x7f997813ab20 2024-04-19T01:44:45.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:45.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f997cfcc640 10 monclient(hunting): authenticate will time out at 1071.555881s 2024-04-19T01:44:45.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.733+0000 7f9975d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99780a52d0 0x7f99781351c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:45.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9975d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99780a52d0 0x7f99781351c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54200/0 (socket says 172.21.15.62:54200) 2024-04-19T01:44:45.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9975d9b640 1 -- 172.21.15.62:0/142615893 learned_addr learned my addr 172.21.15.62:0/142615893 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:44:45.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9975d9b640 10 monclient(hunting): get_auth_request con 0x7f99780a52d0 auth_method 0 2024-04-19T01:44:45.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9976d9d640 1 --2- 172.21.15.62:0/142615893 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f997813ab20 0x7f997811bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:45.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9975d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:45.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9975d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:45.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9975d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:45.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9976d9d640 10 monclient(hunting): get_auth_request con 0x7f997813ab20 auth_method 0 2024-04-19T01:44:45.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9976d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:45.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9976d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:45.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9976d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:45.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9975d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:45.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9975d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:45.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9975d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:45.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9976d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:45.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9976d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:45.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9976d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:45.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f997659c640 1 --2- 172.21.15.62:0/142615893 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99780a47e0 0x7f99781cd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:45.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f997659c640 10 monclient(hunting): get_auth_request con 0x7f99780a47e0 auth_method 0 2024-04-19T01:44:45.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f997659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:45.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f997659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:45.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f997659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:45.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9976d9d640 10 monclient(hunting): handle_auth_done global_id 4772 payload 824 2024-04-19T01:44:45.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9976d9d640 1 -- 172.21.15.62:0/142615893 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99780a47e0 msgr2=0x7f99781cd230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:45.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9976d9d640 1 --2- 172.21.15.62:0/142615893 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99780a47e0 0x7f99781cd230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9976d9d640 1 -- 172.21.15.62:0/142615893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99780a52d0 msgr2=0x7f99781351c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:45.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9976d9d640 1 --2- 172.21.15.62:0/142615893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99780a52d0 0x7f99781351c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9976d9d640 10 monclient: _finish_hunting 0 2024-04-19T01:44:45.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9976d9d640 1 monclient: found mon.noname-c 2024-04-19T01:44:45.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9976d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:45.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9976d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:44:45.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9976d9d640 1 -- 172.21.15.62:0/142615893 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f997811fc50 con 0x7f997813ab20 2024-04-19T01:44:45.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9975d9b640 1 --2- 172.21.15.62:0/142615893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99780a52d0 0x7f99781351c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:44:45.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9976d9d640 1 -- 172.21.15.62:0/142615893 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f997813ab20 msgr2=0x7f997811bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:45.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9976d9d640 1 --2- 172.21.15.62:0/142615893 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f997813ab20 0x7f997811bf20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9970007dc0 tx=0x7f997000e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:45.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9966ffd640 1 -- 172.21.15.62:0/142615893 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9970019070 con 0x7f997813ab20 2024-04-19T01:44:45.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9966ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:45.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9966ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:44:45.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9966ffd640 10 monclient: dump: 2024-04-19T01:44:45.744 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:45.745 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:45.745 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.745 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.745 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:45.745 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:45.745 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:45.745 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:45.745 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:45.745 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:45.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9966ffd640 10 monclient: _finish_auth 0 2024-04-19T01:44:45.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9966ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:44:45.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9966ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:45.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9966ffd640 1 -- 172.21.15.62:0/142615893 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9970003bc0 con 0x7f997813ab20 2024-04-19T01:44:45.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9966ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:45.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9966ffd640 1 -- 172.21.15.62:0/142615893 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9970004030 con 0x7f997813ab20 2024-04-19T01:44:45.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f997cfcc640 5 monclient: authenticate success, global_id 4772 2024-04-19T01:44:45.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f997cfcc640 10 monclient: _renew_subs 2024-04-19T01:44:45.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f997cfcc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:44:45.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f997cfcc640 1 -- 172.21.15.62:0/142615893 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f997813b290 con 0x7f997813ab20 2024-04-19T01:44:45.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9966ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:45.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9974d99640 4 set_mon_vals no callback set 2024-04-19T01:44:45.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9966ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:44:45.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9966ffd640 10 monclient: dump: 2024-04-19T01:44:45.746 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:45.747 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:45.747 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.747 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:45.747 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:45.747 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:45.747 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:45.747 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:45.748 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:45.748 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:45.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f997cfcc640 10 monclient: _renew_subs 2024-04-19T01:44:45.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f997cfcc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:44:45.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f997cfcc640 1 -- 172.21.15.62:0/142615893 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f997811f970 con 0x7f997813ab20 2024-04-19T01:44:45.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9966ffd640 1 -- 172.21.15.62:0/142615893 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f997001c070 con 0x7f997813ab20 2024-04-19T01:44:45.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9964ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:45.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9964ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:45.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9964ff9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:44:45.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.737+0000 7f9964ff9640 1 -- 172.21.15.62:0/142615893 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9950005090 con 0x7f997813ab20 2024-04-19T01:44:45.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.741+0000 7f9966ffd640 1 --2- 172.21.15.62:0/142615893 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f9958036420 0x7f99580388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:45.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.741+0000 7f9966ffd640 1 -- 172.21.15.62:0/142615893 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(115..115 src has 1..115) ==== 4738+0+0 (secure 0 0 0) 0x7f99700492b0 con 0x7f997813ab20 2024-04-19T01:44:45.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.741+0000 7f997659c640 1 --2- 172.21.15.62:0/142615893 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f9958036420 0x7f99580388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:45.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.741+0000 7f997659c640 10 monclient: get_auth_request con 0x7f9958036420 auth_method 0 2024-04-19T01:44:45.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.741+0000 7f997659c640 1 --2- 172.21.15.62:0/142615893 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f9958036420 0x7f99580388e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9968004e30 tx=0x7f9968009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:45.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.745+0000 7f9966ffd640 1 -- 172.21.15.62:0/142615893 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f997001ce30 con 0x7f997813ab20 2024-04-19T01:44:45.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.745+0000 7f9966ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:45.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.745+0000 7f9966ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:44:45.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.965+0000 7f9964ff9640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-19T01:44:45.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.965+0000 7f9964ff9640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-19T01:44:45.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.965+0000 7f9964ff9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:44:45.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.965+0000 7f9964ff9640 1 -- 172.21.15.62:0/142615893 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f9950005b90 con 0x7f997813ab20 2024-04-19T01:44:45.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.969+0000 7f9966ffd640 1 -- 172.21.15.62:0/142615893 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f997001d730 con 0x7f997813ab20 2024-04-19T01:44:45.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.969+0000 7f9966ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-19T01:44:45.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.969+0000 7f9966ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:44:45.967 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.969+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:44:45.967 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.969+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=6,config=2,health=11,kv=10,logm=71} crc {auth=1086370249,config=3541143443,health=131078203,kv=3416605158,logm=1594464777}) 2024-04-19T01:44:45.967 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.969+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=7,config=2,health=11,kv=11,logm=69} crc {auth=4000456799,config=3541143443,health=1571192044,kv=3416605158,logm=90773261}) 2024-04-19T01:44:45.967 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.969+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:44:45.968 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.969+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=6,config=2,health=11,kv=10,logm=71} crc {auth=1086370249,config=3541143443,health=131078203,kv=3416605158,logm=1594464777}) 2024-04-19T01:44:45.968 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.969+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=8,config=2,health=12,kv=10,logm=68} crc {auth=643337803,config=3541143443,health=219129265,kv=3580296110,logm=4074909218}) 2024-04-19T01:44:45.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.969+0000 7f9964ff9640 1 -- 172.21.15.62:0/142615893 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f9958036420 msgr2=0x7f99580388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:45.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.969+0000 7f9964ff9640 1 --2- 172.21.15.62:0/142615893 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f9958036420 0x7f99580388e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9968004e30 tx=0x7f9968009040 comp rx=0 tx=0).stop 2024-04-19T01:44:45.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.973+0000 7f9964ff9640 10 monclient: shutdown 2024-04-19T01:44:45.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.973+0000 7f9964ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:45.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.973+0000 7f9964ff9640 1 -- 172.21.15.62:0/142615893 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f997813ab20 msgr2=0x7f997811bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:45.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.973+0000 7f9964ff9640 1 --2- 172.21.15.62:0/142615893 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f997813ab20 0x7f997811bf20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9970007dc0 tx=0x7f997000e4b0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.973+0000 7f9964ff9640 1 -- 172.21.15.62:0/142615893 shutdown_connections 2024-04-19T01:44:45.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.973+0000 7f9964ff9640 1 --2- 172.21.15.62:0/142615893 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f9958036420 0x7f99580388e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.973+0000 7f9964ff9640 1 --2- 172.21.15.62:0/142615893 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f997813ab20 0x7f997811bf20 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.973+0000 7f9964ff9640 1 --2- 172.21.15.62:0/142615893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99780a52d0 0x7f99781351c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.973+0000 7f9964ff9640 1 --2- 172.21.15.62:0/142615893 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99780a47e0 0x7f99781cd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:45.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.973+0000 7f9964ff9640 1 -- 172.21.15.62:0/142615893 >> 172.21.15.62:0/142615893 conn(0x7f99780a23d0 msgr2=0x7f99780a11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:45.970 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.973+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:44:45.970 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.973+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1022405765}) 2024-04-19T01:44:45.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.973+0000 7f9964ff9640 1 -- 172.21.15.62:0/142615893 shutdown_connections 2024-04-19T01:44:45.970 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.973+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2857463235}) 2024-04-19T01:44:45.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:45.973+0000 7f9964ff9640 1 -- 172.21.15.62:0/142615893 wait complete. 2024-04-19T01:44:45.972 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.973+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:44:45.972 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.973+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1022405765}) 2024-04-19T01:44:45.972 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.973+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=762367000}) 2024-04-19T01:44:45.972 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.973+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:44:45.972 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.973+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3102504970}) 2024-04-19T01:44:45.972 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.973+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2952615341}) 2024-04-19T01:44:45.972 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.973+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:44:45.972 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.973+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3102504970}) 2024-04-19T01:44:45.972 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.973+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3360692865}) 2024-04-19T01:44:45.973 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.977+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:44:45.973 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.977+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4247806821}) 2024-04-19T01:44:45.973 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.977+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2675518931}) 2024-04-19T01:44:45.974 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.977+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:44:45.974 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.977+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4247806821}) 2024-04-19T01:44:45.974 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.977+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2376179418}) 2024-04-19T01:44:45.975 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.977+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:44:45.975 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.977+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1494095624}) 2024-04-19T01:44:45.975 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.977+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2242574085}) 2024-04-19T01:44:45.975 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.977+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:44:45.975 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.977+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1494095624}) 2024-04-19T01:44:45.976 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.977+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2335220795}) 2024-04-19T01:44:45.977 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.981+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:44:45.978 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.981+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3905319197}) 2024-04-19T01:44:45.978 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.981+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3195940286}) 2024-04-19T01:44:45.978 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.981+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:44:45.978 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.981+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3905319197}) 2024-04-19T01:44:45.978 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.981+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2290050382}) 2024-04-19T01:44:45.980 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.981+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:44:45.980 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.981+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=45,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=13,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=6} crc {logm=3964083469,mdsmap=2840952586,mgr=3717952716,mgr_command_descs=3636332126,mgrstat=2739737096,mon_config_key=18691498,monmap=1260512061,osd_metadata=2171542278,osd_pg_creating=3712311844,osdmap=3686184154}) 2024-04-19T01:44:45.980 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.981+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=45,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=6} crc {logm=3755257680,mdsmap=3764978221,mgr=3717952716,mgr_command_descs=3636332126,mgr_metadata=230233602,mgrstat=3540976548,mon_config_key=18691498,monmap=1260512061,osd_metadata=2171542278,osd_pg_creating=3712311844,osdmap=3686184154}) 2024-04-19T01:44:45.980 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.981+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:44:45.981 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.981+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=45,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=13,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=6} crc {logm=3964083469,mdsmap=2840952586,mgr=3717952716,mgr_command_descs=3636332126,mgrstat=2739737096,mon_config_key=18691498,monmap=1260512061,osd_metadata=2171542278,osd_pg_creating=3712311844,osdmap=3686184154}) 2024-04-19T01:44:45.981 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.981+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=44,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=13,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=8} crc {logm=702992284,mdsmap=2840952586,mgr=3717952716,mgr_command_descs=3636332126,mgr_metadata=230233602,mgrstat=2739737096,mon_config_key=3132169225,monmap=405663308,osd_metadata=2171542278,osd_pg_creating=3712311844,osdmap=3015544999}) 2024-04-19T01:44:45.983 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.985+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:44:45.983 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.985+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=304876983}) 2024-04-19T01:44:45.983 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.985+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4108651698}) 2024-04-19T01:44:45.983 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.985+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:44:45.983 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.985+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=304876983}) 2024-04-19T01:44:45.983 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.985+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1579628649}) 2024-04-19T01:44:45.985 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-19T01:44:45.986 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.989+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:44:45.986 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.989+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=71} crc {osdmap=3301228637}) 2024-04-19T01:44:45.986 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.989+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=73} crc {osdmap=1079686508}) 2024-04-19T01:44:45.986 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.989+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:44:45.986 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.989+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=71} crc {osdmap=3301228637}) 2024-04-19T01:44:45.987 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:44:45.989+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=73} crc {osdmap=3788795135}) 2024-04-19T01:44:46.134 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28874]: starting. 2024-04-19T01:44:46.135 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28874]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-28872 2024-04-19T01:44:46.135 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28874]: shutting down. 2024-04-19T01:44:46.986 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-19T01:44:46.987 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:44:47.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6f2687640 10 monclient: get_monmap_and_config 2024-04-19T01:44:47.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6f2687640 10 monclient: build_initial_monmap 2024-04-19T01:44:47.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6f2687640 10 monclient: monmap: 2024-04-19T01:44:47.118 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:47.118 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:47.118 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:44:47.123608+0000 2024-04-19T01:44:47.119 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:44:47.123608+0000 2024-04-19T01:44:47.119 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:47.119 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:47.119 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:44:47.119 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:44:47.119 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:44:47.119 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:44:47.119 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:44:47.119 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:47.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6f2687640 1 Processor -- start 2024-04-19T01:44:47.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6f2687640 1 -- start start 2024-04-19T01:44:47.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6f2687640 10 monclient: init 2024-04-19T01:44:47.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6f2687640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:47.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6f2687640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-04-19T01:44:47.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6f2687640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd6ec137780 addr v1:172.21.15.71:6789/0 2024-04-19T01:44:47.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6f2687640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd6ec0a60b0 0x7fd6ec13c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:47.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6f2687640 10 monclient(hunting): picked mon.noname-a con 0x7fd6ec0a60b0 addr v2:172.21.15.62:3300/0 2024-04-19T01:44:47.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6f2687640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd6ec13cc60 addr v1:172.21.15.62:6789/0 2024-04-19T01:44:47.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6f2687640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6ec09c2e0 con 0x7fd6ec13cc60 2024-04-19T01:44:47.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6f2687640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6ec09c000 con 0x7fd6ec137780 2024-04-19T01:44:47.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6f2687640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:47.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6f2687640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fd6ec09bd20 con 0x7fd6ec0a60b0 2024-04-19T01:44:47.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6f2687640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:47.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6f2687640 10 monclient(hunting): authenticate will time out at 1072.943163s 2024-04-19T01:44:47.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6eb7fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd6ec0a60b0 0x7fd6ec13c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:47.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6eb7fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd6ec0a60b0 0x7fd6ec13c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54210/0 (socket says 172.21.15.62:54210) 2024-04-19T01:44:47.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6f0c24640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fd6ec13cc60 0x7fd6ec13f050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42750/0 (socket says 172.21.15.62:42750) 2024-04-19T01:44:47.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6eb7fe640 1 -- 172.21.15.62:0/1666503316 learned_addr learned my addr 172.21.15.62:0/1666503316 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:44:47.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6eb7fe640 10 monclient(hunting): get_auth_request con 0x7fd6ec0a60b0 auth_method 0 2024-04-19T01:44:47.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:47.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6eb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:47.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:47.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:47.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:47.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:47.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6eb7fe640 10 monclient(hunting): handle_auth_done global_id 14190 payload 274 2024-04-19T01:44:47.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6eb7fe640 1 -- 172.21.15.62:0/1666503316 >> v1:172.21.15.71:6789/0 conn(0x7fd6ec137780 legacy=0x7fd6ec137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:44:47.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6eb7fe640 1 -- 172.21.15.62:0/1666503316 >> v1:172.21.15.62:6789/0 conn(0x7fd6ec13cc60 legacy=0x7fd6ec13f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:44:47.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6eb7fe640 10 monclient: _finish_hunting 0 2024-04-19T01:44:47.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6eb7fe640 1 monclient: found mon.noname-a 2024-04-19T01:44:47.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:47.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6eb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:44:47.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.121+0000 7fd6eb7fe640 1 -- 172.21.15.62:0/1666503316 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6ec138270 con 0x7fd6ec0a60b0 2024-04-19T01:44:47.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.125+0000 7fd6eb7fe640 1 -- 172.21.15.62:0/1666503316 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6ec0a60b0 msgr2=0x7fd6ec13c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:47.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.125+0000 7fd6eb7fe640 1 --2- 172.21.15.62:0/1666503316 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6ec0a60b0 0x7fd6ec13c720 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0009870 tx=0x7fd6e001c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fab7a394f1ce5b61 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:47.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6eaffd640 1 -- 172.21.15.62:0/1666503316 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e00b0070 con 0x7fd6ec0a60b0 2024-04-19T01:44:47.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:47.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6eaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:44:47.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6eaffd640 10 monclient: dump: 2024-04-19T01:44:47.129 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:47.129 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:47.130 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.130 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.130 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:47.130 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:47.130 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:47.130 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:47.130 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:47.131 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:47.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6eaffd640 10 monclient: _finish_auth 0 2024-04-19T01:44:47.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6eaffd640 10 monclient: _check_auth_tickets 2024-04-19T01:44:47.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:47.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 5 monclient: authenticate success, global_id 14190 2024-04-19T01:44:47.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6eaffd640 1 -- 172.21.15.62:0/1666503316 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6e00ab600 con 0x7fd6ec0a60b0 2024-04-19T01:44:47.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:44:47.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6eaffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:47.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 10 monclient: get_monmap_and_config success 2024-04-19T01:44:47.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 4 set_mon_vals no callback set 2024-04-19T01:44:47.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6eaffd640 1 -- 172.21.15.62:0/1666503316 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e00ab9d0 con 0x7fd6ec0a60b0 2024-04-19T01:44:47.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:47.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:44:47.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6eaffd640 10 monclient: dump: 2024-04-19T01:44:47.132 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:47.132 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:47.132 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.132 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.132 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:47.132 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:47.132 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:47.132 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:47.132 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:47.132 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:47.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 10 monclient: shutdown 2024-04-19T01:44:47.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:47.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 1 -- 172.21.15.62:0/1666503316 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6ec0a60b0 msgr2=0x7fd6ec13c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:47.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 1 --2- 172.21.15.62:0/1666503316 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6ec0a60b0 0x7fd6ec13c720 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0009870 tx=0x7fd6e001c2d0 comp rx=0 tx=0).stop 2024-04-19T01:44:47.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 1 -- 172.21.15.62:0/1666503316 shutdown_connections 2024-04-19T01:44:47.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 1 --2- 172.21.15.62:0/1666503316 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6ec0a60b0 0x7fd6ec13c720 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:47.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 1 -- 172.21.15.62:0/1666503316 >> 172.21.15.62:0/1666503316 conn(0x7fd6ec0aa1d0 msgr2=0x7fd6ec0a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:47.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 1 -- 172.21.15.62:0/1666503316 shutdown_connections 2024-04-19T01:44:47.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 1 -- 172.21.15.62:0/1666503316 wait complete. 2024-04-19T01:44:47.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 10 monclient: build_initial_monmap 2024-04-19T01:44:47.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 10 monclient: monmap: 2024-04-19T01:44:47.133 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:47.133 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:44:47.133 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:44:47.133 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:44:47.133 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:47.133 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:47.133 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:44:47.133 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:44:47.134 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:44:47.134 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:47.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 1 Processor -- start 2024-04-19T01:44:47.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 1 -- start start 2024-04-19T01:44:47.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 10 monclient: init 2024-04-19T01:44:47.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:47.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-19T01:44:47.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6ec0a60b0 0x7fd6ec1cf610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:47.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 10 monclient(hunting): picked mon.noname-b con 0x7fd6ec0a60b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:44:47.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6ec137780 0x7fd6ec136410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:47.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 10 monclient(hunting): picked mon.noname-c con 0x7fd6ec137780 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:44:47.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6ec13cc60 0x7fd6ec09e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:47.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 10 monclient(hunting): picked mon.noname-a con 0x7fd6ec13cc60 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:44:47.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:47.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd6ec143320 con 0x7fd6ec13cc60 2024-04-19T01:44:47.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:47.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd6ec1431a0 con 0x7fd6ec0a60b0 2024-04-19T01:44:47.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:47.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fd6ec1434a0 con 0x7fd6ec137780 2024-04-19T01:44:47.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:47.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f2687640 10 monclient(hunting): authenticate will time out at 1072.950180s 2024-04-19T01:44:47.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6eb7fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6ec137780 0x7fd6ec136410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:47.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f0c24640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6ec13cc60 0x7fd6ec09e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:47.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6eb7fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6ec137780 0x7fd6ec136410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:51512/0 (socket says 172.21.15.62:51512) 2024-04-19T01:44:47.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f0c24640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6ec13cc60 0x7fd6ec09e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54218/0 (socket says 172.21.15.62:54218) 2024-04-19T01:44:47.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6eb7fe640 1 -- 172.21.15.62:0/3106972396 learned_addr learned my addr 172.21.15.62:0/3106972396 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:44:47.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6eb7fe640 10 monclient(hunting): get_auth_request con 0x7fd6ec137780 auth_method 0 2024-04-19T01:44:47.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:47.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6eb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:47.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:47.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f0c24640 10 monclient(hunting): get_auth_request con 0x7fd6ec13cc60 auth_method 0 2024-04-19T01:44:47.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f0c24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:47.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f0c24640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:47.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f0c24640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:47.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:47.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:47.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:47.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f0c24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:47.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f0c24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:47.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f0c24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:47.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6ebfff640 1 --2- 172.21.15.62:0/3106972396 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6ec0a60b0 0x7fd6ec1cf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:47.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f0c24640 10 monclient(hunting): handle_auth_done global_id 14193 payload 824 2024-04-19T01:44:47.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f0c24640 1 -- 172.21.15.62:0/3106972396 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6ec137780 msgr2=0x7fd6ec136410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:47.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f0c24640 1 --2- 172.21.15.62:0/3106972396 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6ec137780 0x7fd6ec136410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:47.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f0c24640 1 -- 172.21.15.62:0/3106972396 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6ec0a60b0 msgr2=0x7fd6ec1cf610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:47.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f0c24640 1 --2- 172.21.15.62:0/3106972396 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6ec0a60b0 0x7fd6ec1cf610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:47.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f0c24640 10 monclient: _finish_hunting 0 2024-04-19T01:44:47.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f0c24640 1 monclient: found mon.noname-a 2024-04-19T01:44:47.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f0c24640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:47.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f0c24640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:44:47.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6f0c24640 1 -- 172.21.15.62:0/3106972396 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6ec0a0e70 con 0x7fd6ec13cc60 2024-04-19T01:44:47.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6eb7fe640 1 --2- 172.21.15.62:0/3106972396 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6ec137780 0x7fd6ec136410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:44:47.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6ebfff640 10 monclient: get_auth_request con 0x7fd6ec0a60b0 auth_method 0 2024-04-19T01:44:47.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.129+0000 7fd6ebfff640 1 --2- 172.21.15.62:0/3106972396 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6ec0a60b0 0x7fd6ec1cf610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:44:47.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.133+0000 7fd6f0c24640 1 -- 172.21.15.62:0/3106972396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6ec13cc60 msgr2=0x7fd6ec09e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:47.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.133+0000 7fd6f0c24640 1 --2- 172.21.15.62:0/3106972396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6ec13cc60 0x7fd6ec09e940 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc00c750 tx=0x7fd6dc00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:47.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.133+0000 7fd6e8ff9640 1 -- 172.21.15.62:0/3106972396 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc017070 con 0x7fd6ec13cc60 2024-04-19T01:44:47.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.133+0000 7fd6e8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:47.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.133+0000 7fd6e8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:44:47.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.133+0000 7fd6e8ff9640 10 monclient: dump: 2024-04-19T01:44:47.140 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:47.140 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:47.140 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.141 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.141 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:47.141 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:47.141 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:47.141 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:47.141 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:47.141 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:47.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.133+0000 7fd6e8ff9640 10 monclient: _finish_auth 0 2024-04-19T01:44:47.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.133+0000 7fd6e8ff9640 10 monclient: _check_auth_tickets 2024-04-19T01:44:47.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.133+0000 7fd6e8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:47.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.133+0000 7fd6e8ff9640 1 -- 172.21.15.62:0/3106972396 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6dc00a640 con 0x7fd6ec13cc60 2024-04-19T01:44:47.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.133+0000 7fd6e8ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:47.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.133+0000 7fd6e8ff9640 1 -- 172.21.15.62:0/3106972396 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc00aa30 con 0x7fd6ec13cc60 2024-04-19T01:44:47.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.133+0000 7fd6e8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:47.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.133+0000 7fd6e8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:44:47.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.133+0000 7fd6e8ff9640 10 monclient: dump: 2024-04-19T01:44:47.143 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:47.143 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:47.143 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.143 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.144 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:47.144 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:47.144 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:47.144 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:47.144 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:47.144 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:47.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.133+0000 7fd6f2687640 5 monclient: authenticate success, global_id 14193 2024-04-19T01:44:47.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.133+0000 7fd6f2687640 10 monclient: _renew_subs 2024-04-19T01:44:47.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.133+0000 7fd6f2687640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:44:47.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.133+0000 7fd6f2687640 1 -- 172.21.15.62:0/3106972396 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6ec0a3420 con 0x7fd6ec13cc60 2024-04-19T01:44:47.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.133+0000 7fd6e9ffb640 4 set_mon_vals no callback set 2024-04-19T01:44:47.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.133+0000 7fd6f2687640 10 monclient: _renew_subs 2024-04-19T01:44:47.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.133+0000 7fd6f2687640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:44:47.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.133+0000 7fd6f2687640 1 -- 172.21.15.62:0/3106972396 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6ec0a25a0 con 0x7fd6ec13cc60 2024-04-19T01:44:47.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.133+0000 7fd6f2687640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:47.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.133+0000 7fd6f2687640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:47.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.133+0000 7fd6f2687640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:44:47.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.133+0000 7fd6f2687640 1 -- 172.21.15.62:0/3106972396 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6ec068ab0 con 0x7fd6ec13cc60 2024-04-19T01:44:47.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.133+0000 7fd6e8ff9640 1 -- 172.21.15.62:0/3106972396 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fd6dc01a070 con 0x7fd6ec13cc60 2024-04-19T01:44:47.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.133+0000 7fd6e8ff9640 1 --2- 172.21.15.62:0/3106972396 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fd6d8036470 0x7fd6d8038930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:47.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.133+0000 7fd6e8ff9640 1 -- 172.21.15.62:0/3106972396 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 4738+0+0 (secure 0 0 0) 0x7fd6dc048b00 con 0x7fd6ec13cc60 2024-04-19T01:44:47.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.133+0000 7fd6ebfff640 1 --2- 172.21.15.62:0/3106972396 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fd6d8036470 0x7fd6d8038930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:47.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.133+0000 7fd6ebfff640 10 monclient: get_auth_request con 0x7fd6d8036470 auth_method 0 2024-04-19T01:44:47.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.137+0000 7fd6ebfff640 1 --2- 172.21.15.62:0/3106972396 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fd6d8036470 0x7fd6d8038930 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4003e40 tx=0x7fd6e4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:47.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.141+0000 7fd6e8ff9640 1 -- 172.21.15.62:0/3106972396 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd6dc01a360 con 0x7fd6ec13cc60 2024-04-19T01:44:47.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.141+0000 7fd6e8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:47.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.141+0000 7fd6e8ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:44:47.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.365+0000 7fd6f2687640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:44:47.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.365+0000 7fd6f2687640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:44:47.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.365+0000 7fd6f2687640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:44:47.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.365+0000 7fd6f2687640 1 -- 172.21.15.62:0/3106972396 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd6ec1d0490 con 0x7fd6ec13cc60 2024-04-19T01:44:47.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.365+0000 7fd6e8ff9640 1 -- 172.21.15.62:0/3106972396 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fd6dc00adf0 con 0x7fd6ec13cc60 2024-04-19T01:44:47.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.365+0000 7fd6e8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:44:47.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.365+0000 7fd6e8ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:44:47.365 INFO:teuthology.orchestra.run.smithi062.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","modified":"2024-04-19T01:41:38.497795Z","created":"2024-04-19T01:41:38.497795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:44:47.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.369+0000 7fd6f2687640 1 -- 172.21.15.62:0/3106972396 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fd6d8036470 msgr2=0x7fd6d8038930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:47.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.369+0000 7fd6f2687640 1 --2- 172.21.15.62:0/3106972396 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fd6d8036470 0x7fd6d8038930 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4003e40 tx=0x7fd6e4009040 comp rx=0 tx=0).stop 2024-04-19T01:44:47.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.369+0000 7fd6f2687640 10 monclient: shutdown 2024-04-19T01:44:47.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.369+0000 7fd6f2687640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:47.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.369+0000 7fd6f2687640 1 -- 172.21.15.62:0/3106972396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6ec13cc60 msgr2=0x7fd6ec09e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:47.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.369+0000 7fd6f2687640 1 --2- 172.21.15.62:0/3106972396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6ec13cc60 0x7fd6ec09e940 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc00c750 tx=0x7fd6dc00cc10 comp rx=0 tx=0).stop 2024-04-19T01:44:47.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.369+0000 7fd6f2687640 1 -- 172.21.15.62:0/3106972396 shutdown_connections 2024-04-19T01:44:47.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.369+0000 7fd6f2687640 1 --2- 172.21.15.62:0/3106972396 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fd6d8036470 0x7fd6d8038930 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:47.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.369+0000 7fd6f2687640 1 --2- 172.21.15.62:0/3106972396 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fd6ec137780 0x7fd6ec136410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:47.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.369+0000 7fd6f2687640 1 --2- 172.21.15.62:0/3106972396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6ec13cc60 0x7fd6ec09e940 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:47.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.369+0000 7fd6f2687640 1 --2- 172.21.15.62:0/3106972396 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd6ec0a60b0 0x7fd6ec1cf610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:47.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.369+0000 7fd6f2687640 1 -- 172.21.15.62:0/3106972396 >> 172.21.15.62:0/3106972396 conn(0x7fd6ec0aa1d0 msgr2=0x7fd6ec0a79d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:47.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.369+0000 7fd6f2687640 1 -- 172.21.15.62:0/3106972396 shutdown_connections 2024-04-19T01:44:47.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.369+0000 7fd6f2687640 1 -- 172.21.15.62:0/3106972396 wait complete. 2024-04-19T01:44:47.383 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-19T01:44:47.384 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-19T01:44:47.384 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-19T01:44:47.384 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-19T01:44:47.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22eaae3640 10 monclient: get_monmap_and_config 2024-04-19T01:44:47.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22eaae3640 10 monclient: build_initial_monmap 2024-04-19T01:44:47.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22eaae3640 10 monclient: monmap: 2024-04-19T01:44:47.518 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:47.518 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:47.518 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:44:47.522898+0000 2024-04-19T01:44:47.518 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:44:47.522898+0000 2024-04-19T01:44:47.518 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:47.518 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:47.518 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:44:47.518 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:44:47.518 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:44:47.518 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:44:47.518 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:44:47.518 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:47.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22eaae3640 1 Processor -- start 2024-04-19T01:44:47.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22eaae3640 1 -- start start 2024-04-19T01:44:47.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22eaae3640 10 monclient: init 2024-04-19T01:44:47.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22eaae3640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:47.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22eaae3640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-04-19T01:44:47.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22eaae3640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f22e40a4ae0 0x7f22e40a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:47.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22eaae3640 10 monclient(hunting): picked mon.noname-a con 0x7f22e40a4ae0 addr v2:172.21.15.62:3300/0 2024-04-19T01:44:47.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22eaae3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f22e40a5400 addr v1:172.21.15.71:6789/0 2024-04-19T01:44:47.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22eaae3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f22e40a45d0 addr v1:172.21.15.62:6789/0 2024-04-19T01:44:47.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22eaae3640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22e413c5e0 con 0x7f22e40a45d0 2024-04-19T01:44:47.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22eaae3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22e4071800 con 0x7f22e40a5400 2024-04-19T01:44:47.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22eaae3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:47.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22eaae3640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f22e406dd80 con 0x7f22e40a4ae0 2024-04-19T01:44:47.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22eaae3640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:47.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22eaae3640 10 monclient(hunting): authenticate will time out at 1073.342394s 2024-04-19T01:44:47.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22e9080640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f22e40a45d0 0x7f22e413bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42754/0 (socket says 172.21.15.62:42754) 2024-04-19T01:44:47.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22e9080640 1 -- 172.21.15.62:0/253599665 learned_addr learned my addr 172.21.15.62:0/253599665 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:44:47.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22e887f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f22e40a4ae0 0x7f22e40a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:47.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22e887f640 10 monclient(hunting): get_auth_request con 0x7f22e40a4ae0 auth_method 0 2024-04-19T01:44:47.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22e887f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:47.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22e887f640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:47.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22e887f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:47.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22e887f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:47.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22e887f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:47.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22e887f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:47.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22e887f640 10 monclient(hunting): handle_auth_done global_id 14196 payload 274 2024-04-19T01:44:47.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22e887f640 1 -- 172.21.15.62:0/253599665 >> v1:172.21.15.71:6789/0 conn(0x7f22e40a5400 legacy=0x7f22e413b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:44:47.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22e887f640 1 -- 172.21.15.62:0/253599665 >> v1:172.21.15.62:6789/0 conn(0x7f22e40a45d0 legacy=0x7f22e413bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:44:47.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22e887f640 10 monclient: _finish_hunting 0 2024-04-19T01:44:47.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22e887f640 1 monclient: found mon.noname-a 2024-04-19T01:44:47.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22e887f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:47.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22e887f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:44:47.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.521+0000 7f22e887f640 1 -- 172.21.15.62:0/253599665 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22e406e060 con 0x7f22e40a4ae0 2024-04-19T01:44:47.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.525+0000 7f22e887f640 1 -- 172.21.15.62:0/253599665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e40a4ae0 msgr2=0x7f22e40a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:47.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.525+0000 7f22e887f640 1 --2- 172.21.15.62:0/253599665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e40a4ae0 0x7f22e40a4ec0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f22d40097c0 tx=0x7f22d401c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f23d6f0c759949ba server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:47.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.525+0000 7f22e37fe640 1 -- 172.21.15.62:0/253599665 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d40b0070 con 0x7f22e40a4ae0 2024-04-19T01:44:47.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.525+0000 7f22e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:47.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.525+0000 7f22e37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:44:47.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.525+0000 7f22e37fe640 10 monclient: dump: 2024-04-19T01:44:47.525 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:47.525 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:47.525 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.525 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.525 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:47.525 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:47.525 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:47.525 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:47.525 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:47.525 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:47.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.525+0000 7f22e37fe640 10 monclient: _finish_auth 0 2024-04-19T01:44:47.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.525+0000 7f22e37fe640 10 monclient: _check_auth_tickets 2024-04-19T01:44:47.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.525+0000 7f22e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:47.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.525+0000 7f22e37fe640 1 -- 172.21.15.62:0/253599665 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22d40ab600 con 0x7f22e40a4ae0 2024-04-19T01:44:47.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.525+0000 7f22eaae3640 5 monclient: authenticate success, global_id 14196 2024-04-19T01:44:47.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.525+0000 7f22eaae3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:44:47.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.525+0000 7f22e37fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:47.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.525+0000 7f22eaae3640 10 monclient: get_monmap_and_config success 2024-04-19T01:44:47.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.525+0000 7f22eaae3640 4 set_mon_vals no callback set 2024-04-19T01:44:47.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.525+0000 7f22e37fe640 1 -- 172.21.15.62:0/253599665 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d40ab9d0 con 0x7f22e40a4ae0 2024-04-19T01:44:47.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.525+0000 7f22e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:47.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.525+0000 7f22e37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:44:47.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.525+0000 7f22e37fe640 10 monclient: dump: 2024-04-19T01:44:47.527 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:47.527 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:47.527 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.527 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.527 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:47.527 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:47.527 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:47.527 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:47.527 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:47.527 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:47.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22eaae3640 10 monclient: shutdown 2024-04-19T01:44:47.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22eaae3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:47.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22eaae3640 1 -- 172.21.15.62:0/253599665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e40a4ae0 msgr2=0x7f22e40a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:47.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22eaae3640 1 --2- 172.21.15.62:0/253599665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e40a4ae0 0x7f22e40a4ec0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f22d40097c0 tx=0x7f22d401c360 comp rx=0 tx=0).stop 2024-04-19T01:44:47.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22eaae3640 1 -- 172.21.15.62:0/253599665 shutdown_connections 2024-04-19T01:44:47.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22eaae3640 1 --2- 172.21.15.62:0/253599665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e40a4ae0 0x7f22e40a4ec0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:47.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22eaae3640 1 -- 172.21.15.62:0/253599665 >> 172.21.15.62:0/253599665 conn(0x7f22e40aa1d0 msgr2=0x7f22e40a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:47.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22eaae3640 1 -- 172.21.15.62:0/253599665 shutdown_connections 2024-04-19T01:44:47.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22eaae3640 1 -- 172.21.15.62:0/253599665 wait complete. 2024-04-19T01:44:47.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22eaae3640 10 monclient: build_initial_monmap 2024-04-19T01:44:47.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22eaae3640 10 monclient: monmap: 2024-04-19T01:44:47.538 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:47.538 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:44:47.538 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:44:47.538 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:44:47.538 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:47.538 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:47.538 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:44:47.538 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:44:47.538 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:44:47.538 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:47.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22eaae3640 1 Processor -- start 2024-04-19T01:44:47.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22eaae3640 1 -- start start 2024-04-19T01:44:47.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22eaae3640 10 monclient: init 2024-04-19T01:44:47.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22eaae3640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:47.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22eaae3640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-19T01:44:47.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22eaae3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f22e40a45d0 0x7f22e41d3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:47.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22eaae3640 10 monclient(hunting): picked mon.noname-c con 0x7f22e40a45d0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:44:47.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22eaae3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e40a4ae0 0x7f22e4143bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:47.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22eaae3640 10 monclient(hunting): picked mon.noname-a con 0x7f22e40a4ae0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:44:47.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22eaae3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f22e40a5400 0x7f22e41398c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:47.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22eaae3640 10 monclient(hunting): picked mon.noname-b con 0x7f22e40a5400 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:44:47.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22eaae3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:47.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22eaae3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f22e4147950 con 0x7f22e40a4ae0 2024-04-19T01:44:47.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22eaae3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:47.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22eaae3640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f22e40a63e0 con 0x7f22e40a5400 2024-04-19T01:44:47.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22eaae3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:47.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22eaae3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f22e4147ad0 con 0x7f22e40a45d0 2024-04-19T01:44:47.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22eaae3640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:47.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22eaae3640 10 monclient(hunting): authenticate will time out at 1073.349790s 2024-04-19T01:44:47.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e887f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f22e40a45d0 0x7f22e41d3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:47.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e40a4ae0 0x7f22e4143bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:47.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e887f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f22e40a45d0 0x7f22e41d3ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:51516/0 (socket says 172.21.15.62:51516) 2024-04-19T01:44:47.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e887f640 1 -- 172.21.15.62:0/53309717 learned_addr learned my addr 172.21.15.62:0/53309717 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:44:47.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e887f640 10 monclient(hunting): get_auth_request con 0x7f22e40a45d0 auth_method 0 2024-04-19T01:44:47.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e887f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:47.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e887f640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:47.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e887f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:47.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e3fff640 10 monclient(hunting): get_auth_request con 0x7f22e40a4ae0 auth_method 0 2024-04-19T01:44:47.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:47.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:47.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:47.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e887f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:47.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e887f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:47.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e887f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:47.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:47.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:47.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:47.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e9080640 1 --2- 172.21.15.62:0/53309717 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f22e40a5400 0x7f22e41398c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:47.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e9080640 10 monclient(hunting): get_auth_request con 0x7f22e40a5400 auth_method 0 2024-04-19T01:44:47.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e9080640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:47.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e9080640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:47.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e9080640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:47.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e3fff640 10 monclient(hunting): handle_auth_done global_id 14199 payload 824 2024-04-19T01:44:47.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e3fff640 1 -- 172.21.15.62:0/53309717 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f22e40a45d0 msgr2=0x7f22e41d3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:47.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e3fff640 1 --2- 172.21.15.62:0/53309717 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f22e40a45d0 0x7f22e41d3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:47.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e3fff640 1 -- 172.21.15.62:0/53309717 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f22e40a5400 msgr2=0x7f22e41398c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:47.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e3fff640 1 --2- 172.21.15.62:0/53309717 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f22e40a5400 0x7f22e41398c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:47.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e3fff640 10 monclient: _finish_hunting 0 2024-04-19T01:44:47.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e3fff640 1 monclient: found mon.noname-a 2024-04-19T01:44:47.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:47.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:44:47.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e3fff640 1 -- 172.21.15.62:0/53309717 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22e4144750 con 0x7f22e40a4ae0 2024-04-19T01:44:47.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.529+0000 7f22e887f640 1 --2- 172.21.15.62:0/53309717 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f22e40a45d0 0x7f22e41d3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:44:47.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22e3fff640 1 -- 172.21.15.62:0/53309717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e40a4ae0 msgr2=0x7f22e4143bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:47.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22e3fff640 1 --2- 172.21.15.62:0/53309717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e40a4ae0 0x7f22e4143bb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f22d800c410 tx=0x7f22d800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:47.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22e17fa640 1 -- 172.21.15.62:0/53309717 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d8017070 con 0x7f22e40a4ae0 2024-04-19T01:44:47.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22e17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:47.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22e17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:44:47.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22e17fa640 10 monclient: dump: 2024-04-19T01:44:47.543 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:47.543 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:47.543 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.543 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.543 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:47.543 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:47.543 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:47.543 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:47.543 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:47.543 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:47.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22e17fa640 10 monclient: _finish_auth 0 2024-04-19T01:44:47.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22e17fa640 10 monclient: _check_auth_tickets 2024-04-19T01:44:47.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:47.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22e17fa640 1 -- 172.21.15.62:0/53309717 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22d80069e0 con 0x7f22e40a4ae0 2024-04-19T01:44:47.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22eaae3640 5 monclient: authenticate success, global_id 14199 2024-04-19T01:44:47.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22eaae3640 10 monclient: _renew_subs 2024-04-19T01:44:47.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22eaae3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:44:47.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22eaae3640 1 -- 172.21.15.62:0/53309717 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22e40714d0 con 0x7f22e40a4ae0 2024-04-19T01:44:47.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22e17fa640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:47.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22e17fa640 1 -- 172.21.15.62:0/53309717 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d8006dd0 con 0x7f22e40a4ae0 2024-04-19T01:44:47.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22e17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:47.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22e17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:44:47.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22e17fa640 10 monclient: dump: 2024-04-19T01:44:47.544 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:47.544 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:47.545 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.545 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.545 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:47.545 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:47.545 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:47.545 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:47.545 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:47.545 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:47.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22eaae3640 10 monclient: _renew_subs 2024-04-19T01:44:47.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22e2ffd640 4 set_mon_vals no callback set 2024-04-19T01:44:47.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22eaae3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:44:47.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22eaae3640 1 -- 172.21.15.62:0/53309717 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22e4123240 con 0x7f22e40a4ae0 2024-04-19T01:44:47.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22eaae3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:47.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22eaae3640 1 --2- 172.21.15.62:0/53309717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e41209e0 0x7f22e402c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:47.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22eaae3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:47.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22eaae3640 1 -- 172.21.15.62:0/53309717 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f22e4068ab0 con 0x7f22e41209e0 2024-04-19T01:44:47.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22e887f640 1 --2- 172.21.15.62:0/53309717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e41209e0 0x7f22e402c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:47.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22e887f640 10 monclient: get_auth_request con 0x7f22e41209e0 auth_method 0 2024-04-19T01:44:47.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22e887f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:47.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22e887f640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:47.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22e887f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:47.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22e17fa640 1 -- 172.21.15.62:0/53309717 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f22d801a070 con 0x7f22e40a4ae0 2024-04-19T01:44:47.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22e887f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:47.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22e887f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:47.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22e887f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:47.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22e17fa640 1 --2- 172.21.15.62:0/53309717 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f22d0036420 0x7f22d00388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:47.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22e887f640 10 monclient(hunting): handle_auth_done global_id 14202 payload 824 2024-04-19T01:44:47.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22e887f640 1 -- 172.21.15.62:0/53309717 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f22e40a5a40 con 0x7f22e41209e0 2024-04-19T01:44:47.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22e17fa640 1 -- 172.21.15.62:0/53309717 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 4738+0+0 (secure 0 0 0) 0x7f22d80488a0 con 0x7f22e40a4ae0 2024-04-19T01:44:47.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22e9080640 1 --2- 172.21.15.62:0/53309717 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f22d0036420 0x7f22d00388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:47.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.533+0000 7f22e9080640 10 monclient: get_auth_request con 0x7f22d0036420 auth_method 0 2024-04-19T01:44:47.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.537+0000 7f22e887f640 1 -- 172.21.15.62:0/53309717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e41209e0 msgr2=0x7f22e402c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:47.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.537+0000 7f22e887f640 1 --2- 172.21.15.62:0/53309717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e41209e0 0x7f22e402c450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f22d40ae040 tx=0x7f22d4019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:47.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.537+0000 7f22e17fa640 1 -- 172.21.15.62:0/53309717 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d40bc070 con 0x7f22e41209e0 2024-04-19T01:44:47.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.537+0000 7f22e17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:47.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.537+0000 7f22e17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:44:47.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.537+0000 7f22e17fa640 10 monclient: dump: 2024-04-19T01:44:47.547 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:47.547 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:47.547 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.547 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.547 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:47.547 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:47.548 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:47.548 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:47.548 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:47.548 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:47.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.537+0000 7f22e17fa640 1 -- 172.21.15.62:0/53309717 <== mon.0 v2:172.21.15.62:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f22d40b1040 con 0x7f22e41209e0 2024-04-19T01:44:47.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.537+0000 7f22e17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:47.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.537+0000 7f22e17fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:44:47.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.537+0000 7f22e17fa640 1 -- 172.21.15.62:0/53309717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e41209e0 msgr2=0x7f22e402c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:47.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.537+0000 7f22e17fa640 1 --2- 172.21.15.62:0/53309717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e41209e0 0x7f22e402c450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f22d40ae040 tx=0x7f22d4019100 comp rx=0 tx=0).stop 2024-04-19T01:44:47.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.537+0000 7f22e17fa640 1 -- 172.21.15.62:0/53309717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e41209e0 msgr2=0x7f22e402c450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:44:47.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.537+0000 7f22e17fa640 1 --2- 172.21.15.62:0/53309717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e41209e0 0x7f22e402c450 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f22d40ae040 tx=0x7f22d4019100 comp rx=0 tx=0).stop 2024-04-19T01:44:47.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.537+0000 7f22e9080640 1 --2- 172.21.15.62:0/53309717 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f22d0036420 0x7f22d00388e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f22dc007af0 tx=0x7f22dc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:47.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.541+0000 7f22eaae3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:44:47.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.541+0000 7f22eaae3640 1 --2- 172.21.15.62:0/53309717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e4120dc0 0x7f22e402c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:47.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.541+0000 7f22eaae3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:47.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.541+0000 7f22eaae3640 1 -- 172.21.15.62:0/53309717 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f22e413ffd0 con 0x7f22e4120dc0 2024-04-19T01:44:47.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.541+0000 7f22e887f640 1 --2- 172.21.15.62:0/53309717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e4120dc0 0x7f22e402c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:47.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.541+0000 7f22e887f640 10 monclient: get_auth_request con 0x7f22e4120dc0 auth_method 0 2024-04-19T01:44:47.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.541+0000 7f22e887f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:47.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.541+0000 7f22e887f640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:47.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.541+0000 7f22e887f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:47.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.541+0000 7f22e887f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:47.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.541+0000 7f22e887f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:47.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.541+0000 7f22e887f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:47.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.545+0000 7f22e887f640 10 monclient(hunting): handle_auth_done global_id 14205 payload 824 2024-04-19T01:44:47.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.545+0000 7f22e887f640 1 -- 172.21.15.62:0/53309717 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f22e4122d80 con 0x7f22e4120dc0 2024-04-19T01:44:47.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.545+0000 7f22e887f640 1 -- 172.21.15.62:0/53309717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e4120dc0 msgr2=0x7f22e402c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:47.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.545+0000 7f22e887f640 1 --2- 172.21.15.62:0/53309717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e4120dc0 0x7f22e402c990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f22e40637a0 tx=0x7f22d4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:47.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.545+0000 7f22e17fa640 1 -- 172.21.15.62:0/53309717 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d40b0040 con 0x7f22e4120dc0 2024-04-19T01:44:47.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.545+0000 7f22e17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:47.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.545+0000 7f22e17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:44:47.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.545+0000 7f22e17fa640 10 monclient: dump: 2024-04-19T01:44:47.552 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:47.552 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:47.552 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.552 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.552 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:47.552 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:47.552 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:47.553 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:47.553 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:47.553 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:47.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.545+0000 7f22e17fa640 1 -- 172.21.15.62:0/53309717 <== mon.0 v2:172.21.15.62:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4548 (secure 0 0 0) 0x7f22d40aa3a0 con 0x7f22e4120dc0 2024-04-19T01:44:47.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.545+0000 7f22e17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:44:47.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.545+0000 7f22e17fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:44:47.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.545+0000 7f22e17fa640 1 -- 172.21.15.62:0/53309717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e4120dc0 msgr2=0x7f22e402c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:47.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.545+0000 7f22e17fa640 1 --2- 172.21.15.62:0/53309717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e4120dc0 0x7f22e402c990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f22e40637a0 tx=0x7f22d4008040 comp rx=0 tx=0).stop 2024-04-19T01:44:47.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.545+0000 7f22e17fa640 1 -- 172.21.15.62:0/53309717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e4120dc0 msgr2=0x7f22e402c990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:44:47.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.545+0000 7f22e17fa640 1 --2- 172.21.15.62:0/53309717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e4120dc0 0x7f22e402c990 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f22e40637a0 tx=0x7f22d4008040 comp rx=0 tx=0).stop 2024-04-19T01:44:47.553 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-19T01:44:47.554 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "a", 2024-04-19T01:44:47.554 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 0, 2024-04-19T01:44:47.554 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "leader", 2024-04-19T01:44:47.554 INFO:teuthology.orchestra.run.smithi062.stdout: "election_epoch": 28, 2024-04-19T01:44:47.554 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 31107, 2024-04-19T01:44:47.554 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum": [ 2024-04-19T01:44:47.554 INFO:teuthology.orchestra.run.smithi062.stdout: 0, 2024-04-19T01:44:47.554 INFO:teuthology.orchestra.run.smithi062.stdout: 1, 2024-04-19T01:44:47.554 INFO:teuthology.orchestra.run.smithi062.stdout: 2 2024-04-19T01:44:47.554 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:44:47.554 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_age": 5, 2024-04-19T01:44:47.554 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:44:47.554 INFO:teuthology.orchestra.run.smithi062.stdout: "required_con": "2450521708007866372", 2024-04-19T01:44:47.555 INFO:teuthology.orchestra.run.smithi062.stdout: "required_mon": [ 2024-04-19T01:44:47.555 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:44:47.555 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:44:47.555 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:44:47.555 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:44:47.555 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:44:47.555 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:44:47.555 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:44:47.555 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:44:47.555 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:44:47.555 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:44:47.555 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:44:47.555 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:44:47.555 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:44:47.555 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_mon": [ 2024-04-19T01:44:47.556 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:44:47.556 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:44:47.556 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:44:47.556 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:44:47.556 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:44:47.556 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:44:47.556 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:44:47.556 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:44:47.556 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:44:47.556 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:44:47.556 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:44:47.556 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:47.556 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.556 INFO:teuthology.orchestra.run.smithi062.stdout: "outside_quorum": [], 2024-04-19T01:44:47.557 INFO:teuthology.orchestra.run.smithi062.stdout: "extra_probe_peers": [], 2024-04-19T01:44:47.557 INFO:teuthology.orchestra.run.smithi062.stdout: "sync_provider": [], 2024-04-19T01:44:47.557 INFO:teuthology.orchestra.run.smithi062.stdout: "monmap": { 2024-04-19T01:44:47.557 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 1, 2024-04-19T01:44:47.557 INFO:teuthology.orchestra.run.smithi062.stdout: "fsid": "6466e4b6-2520-485f-95b3-9401915fbbad", 2024-04-19T01:44:47.557 INFO:teuthology.orchestra.run.smithi062.stdout: "modified": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:44:47.557 INFO:teuthology.orchestra.run.smithi062.stdout: "created": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:44:47.557 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release": 19, 2024-04-19T01:44:47.557 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release_name": "squid", 2024-04-19T01:44:47.557 INFO:teuthology.orchestra.run.smithi062.stdout: "election_strategy": 1, 2024-04-19T01:44:47.557 INFO:teuthology.orchestra.run.smithi062.stdout: "disallowed_leaders: ": "", 2024-04-19T01:44:47.557 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false, 2024-04-19T01:44:47.557 INFO:teuthology.orchestra.run.smithi062.stdout: "tiebreaker_mon": "", 2024-04-19T01:44:47.557 INFO:teuthology.orchestra.run.smithi062.stdout: "removed_ranks: ": "", 2024-04-19T01:44:47.557 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:44:47.558 INFO:teuthology.orchestra.run.smithi062.stdout: "persistent": [ 2024-04-19T01:44:47.558 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:44:47.558 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:44:47.558 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:44:47.558 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:44:47.558 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:44:47.558 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:44:47.558 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:44:47.558 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:44:47.558 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:44:47.558 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:44:47.558 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:44:47.558 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:44:47.558 INFO:teuthology.orchestra.run.smithi062.stdout: "optional": [] 2024-04-19T01:44:47.559 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.559 INFO:teuthology.orchestra.run.smithi062.stdout: "mons": [ 2024-04-19T01:44:47.559 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.559 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 0, 2024-04-19T01:44:47.559 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "a", 2024-04-19T01:44:47.559 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:44:47.559 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:44:47.559 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.559 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:44:47.559 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3300", 2024-04-19T01:44:47.559 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:47.559 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.559 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.559 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:44:47.559 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789", 2024-04-19T01:44:47.560 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:47.560 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:47.560 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:47.560 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.560 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789/0", 2024-04-19T01:44:47.560 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6789/0", 2024-04-19T01:44:47.560 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:44:47.560 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:44:47.560 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:44:47.560 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.560 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.560 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 1, 2024-04-19T01:44:47.560 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "b", 2024-04-19T01:44:47.560 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:44:47.560 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:44:47.561 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.561 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:44:47.561 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:3300", 2024-04-19T01:44:47.561 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:47.561 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.561 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.561 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:44:47.561 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789", 2024-04-19T01:44:47.561 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:47.561 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:47.561 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:47.561 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.561 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789/0", 2024-04-19T01:44:47.561 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.71:6789/0", 2024-04-19T01:44:47.561 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:44:47.562 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:44:47.562 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:44:47.562 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.562 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.562 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 2, 2024-04-19T01:44:47.562 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "c", 2024-04-19T01:44:47.562 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:44:47.562 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:44:47.562 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.562 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:44:47.562 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3301", 2024-04-19T01:44:47.562 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:47.562 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.562 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.563 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:44:47.563 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790", 2024-04-19T01:44:47.563 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:47.563 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:47.563 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:47.563 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.563 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790/0", 2024-04-19T01:44:47.563 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6790/0", 2024-04-19T01:44:47.563 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:44:47.563 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:44:47.563 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:44:47.563 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:47.563 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:47.563 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.563 INFO:teuthology.orchestra.run.smithi062.stdout: "feature_map": { 2024-04-19T01:44:47.564 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": [ 2024-04-19T01:44:47.564 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.564 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:44:47.564 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:44:47.564 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:44:47.564 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:47.564 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:44:47.564 INFO:teuthology.orchestra.run.smithi062.stdout: "client": [ 2024-04-19T01:44:47.564 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.564 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:44:47.564 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:44:47.564 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 3 2024-04-19T01:44:47.564 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:47.564 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:47.564 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.565 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false 2024-04-19T01:44:47.565 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-19T01:44:47.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.545+0000 7f22eaae3640 1 -- 172.21.15.62:0/53309717 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f22d0036420 msgr2=0x7f22d00388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:47.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.545+0000 7f22eaae3640 1 --2- 172.21.15.62:0/53309717 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f22d0036420 0x7f22d00388e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f22dc007af0 tx=0x7f22dc009040 comp rx=0 tx=0).stop 2024-04-19T01:44:47.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.545+0000 7f22eaae3640 10 monclient: shutdown 2024-04-19T01:44:47.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.545+0000 7f22eaae3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:47.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.545+0000 7f22eaae3640 1 -- 172.21.15.62:0/53309717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e40a4ae0 msgr2=0x7f22e4143bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:47.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.545+0000 7f22eaae3640 1 --2- 172.21.15.62:0/53309717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22e40a4ae0 0x7f22e4143bb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f22d800c410 tx=0x7f22d800c8d0 comp rx=0 tx=0).stop 2024-04-19T01:44:47.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.545+0000 7f22e9080640 1 -- 172.21.15.62:0/53309717 reap_dead start 2024-04-19T01:44:47.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.545+0000 7f22e9080640 1 -- 172.21.15.62:0/53309717 reap_dead start 2024-04-19T01:44:47.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.545+0000 7f22eaae3640 1 -- 172.21.15.62:0/53309717 shutdown_connections 2024-04-19T01:44:47.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.545+0000 7f22eaae3640 1 -- 172.21.15.62:0/53309717 >> 172.21.15.62:0/53309717 conn(0x7f22e40aa1d0 msgr2=0x7f22e40a9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:47.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.545+0000 7f22eaae3640 1 -- 172.21.15.62:0/53309717 shutdown_connections 2024-04-19T01:44:47.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.545+0000 7f22eaae3640 1 -- 172.21.15.62:0/53309717 wait complete. 2024-04-19T01:44:47.567 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-19T01:44:47.567 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-19T01:44:47.567 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-19T01:44:47.624 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:44:47.624 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28907]: starting. 2024-04-19T01:44:47.624 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28907]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-28872 2024-04-19T01:44:47.624 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28907]: created object 0... 2024-04-19T01:44:47.625 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28907]: created object 25... 2024-04-19T01:44:47.625 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28907]: created object 49... 2024-04-19T01:44:47.625 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28907]: finishing. 2024-04-19T01:44:47.625 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28907]: shutting down. 2024-04-19T01:44:47.675 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:44:47.676 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28909]: starting. 2024-04-19T01:44:47.676 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28909]: listing objects. 2024-04-19T01:44:47.676 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28909]: listed object 0... 2024-04-19T01:44:47.676 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28909]: listed object 25... 2024-04-19T01:44:47.676 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28909]: saw 50 objects 2024-04-19T01:44:47.676 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28909]: shutting down. 2024-04-19T01:44:47.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e44fb640 10 monclient: get_monmap_and_config 2024-04-19T01:44:47.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e44fb640 10 monclient: build_initial_monmap 2024-04-19T01:44:47.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e44fb640 10 monclient: monmap: 2024-04-19T01:44:47.682 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:47.682 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:47.682 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:44:47.687827+0000 2024-04-19T01:44:47.683 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:44:47.687827+0000 2024-04-19T01:44:47.683 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:47.683 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:47.683 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:44:47.683 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:44:47.683 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:44:47.683 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:44:47.683 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:44:47.683 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:47.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e44fb640 1 Processor -- start 2024-04-19T01:44:47.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e44fb640 1 -- start start 2024-04-19T01:44:47.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e44fb640 10 monclient: init 2024-04-19T01:44:47.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e44fb640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:47.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e44fb640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-04-19T01:44:47.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e44fb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6dc0a47e0 0x7ff6dc0a4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:47.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e44fb640 10 monclient(hunting): picked mon.noname-b con 0x7ff6dc0a47e0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:44:47.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e44fb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff6dc0a5100 addr v1:172.21.15.71:6789/0 2024-04-19T01:44:47.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e44fb640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff6dc1357c0 0x7ff6dc13d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:47.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e44fb640 10 monclient(hunting): picked mon.noname-c con 0x7ff6dc1357c0 addr v2:172.21.15.71:3300/0 2024-04-19T01:44:47.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e44fb640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6dc09f7c0 con 0x7ff6dc0a5100 2024-04-19T01:44:47.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e44fb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:47.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e44fb640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7ff6dc09bf00 con 0x7ff6dc1357c0 2024-04-19T01:44:47.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e44fb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:47.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e44fb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7ff6dc09ee90 con 0x7ff6dc0a47e0 2024-04-19T01:44:47.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e44fb640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:47.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e44fb640 10 monclient(hunting): authenticate will time out at 1073.507111s 2024-04-19T01:44:47.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e2297640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6dc0a47e0 0x7ff6dc0a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:47.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e2297640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6dc0a47e0 0x7ff6dc0a4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:51526/0 (socket says 172.21.15.62:51526) 2024-04-19T01:44:47.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e2297640 1 -- 172.21.15.62:0/2846229406 learned_addr learned my addr 172.21.15.62:0/2846229406 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:44:47.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e2297640 10 monclient(hunting): get_auth_request con 0x7ff6dc0a47e0 auth_method 0 2024-04-19T01:44:47.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e2297640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:47.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e2297640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:47.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e2297640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:47.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e2297640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:47.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e2297640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:47.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e2297640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:47.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e2a98640 1 --2- 172.21.15.62:0/2846229406 >> v2:172.21.15.71:3300/0 conn(0x7ff6dc1357c0 0x7ff6dc13d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:47.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e2a98640 10 monclient(hunting): get_auth_request con 0x7ff6dc1357c0 auth_method 0 2024-04-19T01:44:47.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e2a98640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:47.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e2a98640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:47.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e2a98640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:47.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e2297640 10 monclient(hunting): handle_auth_done global_id 4796 payload 274 2024-04-19T01:44:47.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e2297640 1 -- 172.21.15.62:0/2846229406 >> v2:172.21.15.71:3300/0 conn(0x7ff6dc1357c0 msgr2=0x7ff6dc13d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:44:47.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e2297640 1 --2- 172.21.15.62:0/2846229406 >> v2:172.21.15.71:3300/0 conn(0x7ff6dc1357c0 0x7ff6dc13d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:47.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e2297640 1 -- 172.21.15.62:0/2846229406 >> v1:172.21.15.71:6789/0 conn(0x7ff6dc0a5100 legacy=0x7ff6dc1350b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:44:47.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e2297640 10 monclient: _finish_hunting 0 2024-04-19T01:44:47.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e2297640 1 monclient: found mon.noname-b 2024-04-19T01:44:47.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e2297640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:47.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e2297640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:44:47.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.685+0000 7ff6e2297640 1 -- 172.21.15.62:0/2846229406 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6dc09c2d0 con 0x7ff6dc0a47e0 2024-04-19T01:44:47.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.689+0000 7ff6e2297640 1 -- 172.21.15.62:0/2846229406 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6dc0a47e0 msgr2=0x7ff6dc0a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:47.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.689+0000 7ff6e2297640 1 --2- 172.21.15.62:0/2846229406 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6dc0a47e0 0x7ff6dc0a4bc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc009870 tx=0x7ff6cc01c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93765ebbb2b3f926 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:47.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.689+0000 7ff6e1295640 1 -- 172.21.15.62:0/2846229406 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc0b0070 con 0x7ff6dc0a47e0 2024-04-19T01:44:47.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.689+0000 7ff6e1295640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:47.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.689+0000 7ff6e1295640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:44:47.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.689+0000 7ff6e1295640 10 monclient: dump: 2024-04-19T01:44:47.693 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:47.693 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:47.693 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.693 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.693 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:47.693 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:47.693 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:47.693 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:47.693 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:47.693 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:47.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.689+0000 7ff6e1295640 10 monclient: _finish_auth 0 2024-04-19T01:44:47.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.689+0000 7ff6e1295640 10 monclient: _check_auth_tickets 2024-04-19T01:44:47.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.689+0000 7ff6e1295640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:47.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.689+0000 7ff6e1295640 1 -- 172.21.15.62:0/2846229406 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6cc0ab600 con 0x7ff6dc0a47e0 2024-04-19T01:44:47.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.689+0000 7ff6e44fb640 5 monclient: authenticate success, global_id 4796 2024-04-19T01:44:47.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.689+0000 7ff6e44fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:44:47.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.689+0000 7ff6e1295640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:47.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.689+0000 7ff6e44fb640 10 monclient: get_monmap_and_config success 2024-04-19T01:44:47.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.689+0000 7ff6e44fb640 4 set_mon_vals no callback set 2024-04-19T01:44:47.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.689+0000 7ff6e1295640 1 -- 172.21.15.62:0/2846229406 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc0ab9d0 con 0x7ff6dc0a47e0 2024-04-19T01:44:47.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.689+0000 7ff6e1295640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:47.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.689+0000 7ff6e1295640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:44:47.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.689+0000 7ff6e1295640 10 monclient: dump: 2024-04-19T01:44:47.695 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:47.695 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:47.695 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.695 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.695 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:47.695 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:47.695 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:47.695 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:47.695 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:47.695 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:47.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e44fb640 10 monclient: shutdown 2024-04-19T01:44:47.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e44fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:47.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e44fb640 1 -- 172.21.15.62:0/2846229406 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6dc0a47e0 msgr2=0x7ff6dc0a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:47.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e44fb640 1 --2- 172.21.15.62:0/2846229406 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6dc0a47e0 0x7ff6dc0a4bc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc009870 tx=0x7ff6cc01c2d0 comp rx=0 tx=0).stop 2024-04-19T01:44:47.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e44fb640 1 -- 172.21.15.62:0/2846229406 shutdown_connections 2024-04-19T01:44:47.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e44fb640 1 --2- 172.21.15.62:0/2846229406 >> v2:172.21.15.71:3300/0 conn(0x7ff6dc1357c0 0x7ff6dc13d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:47.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e44fb640 1 --2- 172.21.15.62:0/2846229406 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6dc0a47e0 0x7ff6dc0a4bc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:47.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e44fb640 1 -- 172.21.15.62:0/2846229406 >> 172.21.15.62:0/2846229406 conn(0x7ff6dc0a23d0 msgr2=0x7ff6dc0a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:47.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e44fb640 1 -- 172.21.15.62:0/2846229406 shutdown_connections 2024-04-19T01:44:47.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e44fb640 1 -- 172.21.15.62:0/2846229406 wait complete. 2024-04-19T01:44:47.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e44fb640 10 monclient: build_initial_monmap 2024-04-19T01:44:47.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e44fb640 10 monclient: monmap: 2024-04-19T01:44:47.696 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:47.696 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:44:47.696 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:44:47.697 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:44:47.697 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:47.697 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:47.697 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:44:47.697 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:44:47.697 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:44:47.697 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:47.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e44fb640 1 Processor -- start 2024-04-19T01:44:47.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e44fb640 1 -- start start 2024-04-19T01:44:47.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e44fb640 10 monclient: init 2024-04-19T01:44:47.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e44fb640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:47.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e44fb640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-19T01:44:47.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e44fb640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6dc0a47e0 0x7ff6dc1cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:47.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e44fb640 10 monclient(hunting): picked mon.noname-a con 0x7ff6dc0a47e0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:44:47.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e44fb640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6dc0a5100 0x7ff6dc13c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:47.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e44fb640 10 monclient(hunting): picked mon.noname-b con 0x7ff6dc0a5100 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:44:47.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e44fb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6dc1357c0 0x7ff6dc13cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:47.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e44fb640 10 monclient(hunting): picked mon.noname-c con 0x7ff6dc1357c0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:44:47.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e44fb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:47.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e44fb640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc09c8e0 con 0x7ff6dc0a47e0 2024-04-19T01:44:47.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e44fb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:47.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e44fb640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc09c760 con 0x7ff6dc0a5100 2024-04-19T01:44:47.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e44fb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:47.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e44fb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7ff6dc140360 con 0x7ff6dc1357c0 2024-04-19T01:44:47.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e44fb640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:47.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e44fb640 10 monclient(hunting): authenticate will time out at 1073.514124s 2024-04-19T01:44:47.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e2297640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6dc0a47e0 0x7ff6dc1cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:47.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e2297640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6dc0a47e0 0x7ff6dc1cd3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54246/0 (socket says 172.21.15.62:54246) 2024-04-19T01:44:47.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e2297640 1 -- 172.21.15.62:0/203564425 learned_addr learned my addr 172.21.15.62:0/203564425 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:44:47.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e2297640 10 monclient(hunting): get_auth_request con 0x7ff6dc0a47e0 auth_method 0 2024-04-19T01:44:47.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e2297640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:47.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e2297640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:47.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e2297640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:47.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e2a98640 1 --2- 172.21.15.62:0/203564425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6dc1357c0 0x7ff6dc13cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:47.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e2a98640 10 monclient(hunting): get_auth_request con 0x7ff6dc1357c0 auth_method 0 2024-04-19T01:44:47.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e2a98640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:47.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e2a98640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:47.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e2a98640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:47.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e2297640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:47.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e2297640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:47.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e2297640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:47.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e2a98640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:47.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e2a98640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:47.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e2a98640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:47.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e1a96640 1 --2- 172.21.15.62:0/203564425 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6dc0a5100 0x7ff6dc13c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:47.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e1a96640 10 monclient(hunting): get_auth_request con 0x7ff6dc0a5100 auth_method 0 2024-04-19T01:44:47.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e1a96640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:47.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e1a96640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:47.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e1a96640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:47.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e2297640 10 monclient(hunting): handle_auth_done global_id 14208 payload 824 2024-04-19T01:44:47.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e2297640 1 -- 172.21.15.62:0/203564425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6dc1357c0 msgr2=0x7ff6dc13cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:47.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e2297640 1 --2- 172.21.15.62:0/203564425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6dc1357c0 0x7ff6dc13cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:47.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e2297640 1 -- 172.21.15.62:0/203564425 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6dc0a5100 msgr2=0x7ff6dc13c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:47.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e2297640 1 --2- 172.21.15.62:0/203564425 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6dc0a5100 0x7ff6dc13c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:47.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e2297640 10 monclient: _finish_hunting 0 2024-04-19T01:44:47.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e2297640 1 monclient: found mon.noname-a 2024-04-19T01:44:47.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e2297640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:47.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e2297640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:44:47.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e2297640 1 -- 172.21.15.62:0/203564425 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6dc139c50 con 0x7ff6dc0a47e0 2024-04-19T01:44:47.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.693+0000 7ff6e2a98640 1 --2- 172.21.15.62:0/203564425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6dc1357c0 0x7ff6dc13cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:44:47.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6e2297640 1 -- 172.21.15.62:0/203564425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6dc0a47e0 msgr2=0x7ff6dc1cd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:47.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6e2297640 1 --2- 172.21.15.62:0/203564425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6dc0a47e0 0x7ff6dc1cd3e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc01cb70 tx=0x7ff6cc01ab10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:47.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6d2ffd640 1 -- 172.21.15.62:0/203564425 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc0ba020 con 0x7ff6dc0a47e0 2024-04-19T01:44:47.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:47.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6d2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:44:47.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6d2ffd640 10 monclient: dump: 2024-04-19T01:44:47.705 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:47.705 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:47.705 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.705 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.705 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:47.705 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:47.705 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:47.705 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:47.705 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:47.705 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:47.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6d2ffd640 10 monclient: _finish_auth 0 2024-04-19T01:44:47.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6d2ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:44:47.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:47.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6d2ffd640 1 -- 172.21.15.62:0/203564425 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6cc0b6420 con 0x7ff6dc0a47e0 2024-04-19T01:44:47.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6d2ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:47.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6d2ffd640 1 -- 172.21.15.62:0/203564425 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc0b0040 con 0x7ff6dc0a47e0 2024-04-19T01:44:47.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6e44fb640 5 monclient: authenticate success, global_id 14208 2024-04-19T01:44:47.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6e44fb640 10 monclient: _renew_subs 2024-04-19T01:44:47.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6e44fb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:44:47.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6e44fb640 1 -- 172.21.15.62:0/203564425 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6dc1387a0 con 0x7ff6dc0a47e0 2024-04-19T01:44:47.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:47.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6e0a94640 4 set_mon_vals no callback set 2024-04-19T01:44:47.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:44:47.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6d2ffd640 10 monclient: dump: 2024-04-19T01:44:47.707 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:47.707 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:47.707 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.707 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.707 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:47.707 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:47.707 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:47.707 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:47.707 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:47.707 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:47.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6e44fb640 10 monclient: _renew_subs 2024-04-19T01:44:47.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6e44fb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:44:47.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6e44fb640 1 -- 172.21.15.62:0/203564425 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6dc1cdb00 con 0x7ff6dc0a47e0 2024-04-19T01:44:47.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6e44fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:47.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6e44fb640 1 --2- 172.21.15.62:0/203564425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6dc1361a0 0x7ff6dc136580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:47.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6e44fb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:47.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6e44fb640 1 -- 172.21.15.62:0/203564425 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7ff6dc0692b0 con 0x7ff6dc1361a0 2024-04-19T01:44:47.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6e1a96640 1 --2- 172.21.15.62:0/203564425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6dc1361a0 0x7ff6dc136580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:47.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6e1a96640 10 monclient: get_auth_request con 0x7ff6dc1361a0 auth_method 0 2024-04-19T01:44:47.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6e1a96640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:47.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6e1a96640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:47.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6e1a96640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:47.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6e1a96640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:47.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6e1a96640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:47.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6e1a96640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:47.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6e2297640 1 --2- 172.21.15.62:0/203564425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6dc0a47e0 0x7ff6dc1cd3e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc01cb70 tx=0x7ff6cc01ab10 comp rx=0 tx=0).queue_received will delay after 1.71349e+09 on 0x7ff6cc0e34d0 osd_map(118..118 src has 1..118) 2024-04-19T01:44:47.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6d2ffd640 1 -- 172.21.15.62:0/203564425 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7ff6cc0bd020 con 0x7ff6dc0a47e0 2024-04-19T01:44:47.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6e1a96640 10 monclient(hunting): handle_auth_done global_id 4802 payload 824 2024-04-19T01:44:47.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6e1a96640 1 -- 172.21.15.62:0/203564425 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff6dc072650 con 0x7ff6dc1361a0 2024-04-19T01:44:47.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6d2ffd640 1 --2- 172.21.15.62:0/203564425 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7ff6c0036450 0x7ff6c0038910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:47.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6e2a98640 1 --2- 172.21.15.62:0/203564425 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7ff6c0036450 0x7ff6c0038910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:47.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.697+0000 7ff6e2a98640 10 monclient: get_auth_request con 0x7ff6c0036450 auth_method 0 2024-04-19T01:44:47.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.701+0000 7ff6e1a96640 1 -- 172.21.15.62:0/203564425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6dc1361a0 msgr2=0x7ff6dc136580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:47.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.701+0000 7ff6e1a96640 1 --2- 172.21.15.62:0/203564425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6dc1361a0 0x7ff6dc136580 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc139e60 tx=0x7ff6d800e8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:47.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.701+0000 7ff6d2ffd640 1 -- 172.21.15.62:0/203564425 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d8007450 con 0x7ff6dc1361a0 2024-04-19T01:44:47.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.701+0000 7ff6d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:47.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.701+0000 7ff6d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:44:47.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.701+0000 7ff6d2ffd640 10 monclient: dump: 2024-04-19T01:44:47.709 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:47.709 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:47.709 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.709 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.709 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:47.710 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:47.710 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:47.710 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:47.710 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:47.710 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:47.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.701+0000 7ff6d2ffd640 1 -- 172.21.15.62:0/203564425 <== mon.2 v2:172.21.15.62:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff6d8007450 con 0x7ff6dc1361a0 2024-04-19T01:44:47.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.701+0000 7ff6d2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:47.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.701+0000 7ff6d2ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:44:47.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.701+0000 7ff6d2ffd640 1 -- 172.21.15.62:0/203564425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6dc1361a0 msgr2=0x7ff6dc136580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:47.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.701+0000 7ff6d2ffd640 1 --2- 172.21.15.62:0/203564425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6dc1361a0 0x7ff6dc136580 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc139e60 tx=0x7ff6d800e8b0 comp rx=0 tx=0).stop 2024-04-19T01:44:47.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.701+0000 7ff6d2ffd640 1 -- 172.21.15.62:0/203564425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6dc1361a0 msgr2=0x7ff6dc136580 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:44:47.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.701+0000 7ff6d2ffd640 1 --2- 172.21.15.62:0/203564425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6dc1361a0 0x7ff6dc136580 secure :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc139e60 tx=0x7ff6d800e8b0 comp rx=0 tx=0).stop 2024-04-19T01:44:47.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.701+0000 7ff6e2a98640 1 --2- 172.21.15.62:0/203564425 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7ff6c0036450 0x7ff6c0038910 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc13a0b0 tx=0x7ff6d4009820 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:47.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.705+0000 7ff6e44fb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:44:47.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.705+0000 7ff6e44fb640 1 --2- 172.21.15.62:0/203564425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6dc136ac0 0x7ff6dc070780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:47.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.705+0000 7ff6e44fb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:47.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.705+0000 7ff6e44fb640 1 -- 172.21.15.62:0/203564425 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7ff6dc06e770 con 0x7ff6dc136ac0 2024-04-19T01:44:47.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.705+0000 7ff6e1a96640 1 --2- 172.21.15.62:0/203564425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6dc136ac0 0x7ff6dc070780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:47.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.705+0000 7ff6e1a96640 10 monclient: get_auth_request con 0x7ff6dc136ac0 auth_method 0 2024-04-19T01:44:47.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.705+0000 7ff6e1a96640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:47.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.705+0000 7ff6e1a96640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:47.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.705+0000 7ff6e1a96640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:47.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.705+0000 7ff6e1a96640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:47.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.705+0000 7ff6e1a96640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:47.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.705+0000 7ff6e1a96640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:47.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.705+0000 7ff6e1a96640 10 monclient(hunting): handle_auth_done global_id 4805 payload 824 2024-04-19T01:44:47.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.705+0000 7ff6e1a96640 1 -- 172.21.15.62:0/203564425 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff6dc06b980 con 0x7ff6dc136ac0 2024-04-19T01:44:47.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.709+0000 7ff6e1a96640 1 -- 172.21.15.62:0/203564425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6dc136ac0 msgr2=0x7ff6dc070780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:47.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.709+0000 7ff6e1a96640 1 --2- 172.21.15.62:0/203564425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6dc136ac0 0x7ff6dc070780 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc139e60 tx=0x7ff6d8003770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:47.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.709+0000 7ff6d2ffd640 1 -- 172.21.15.62:0/203564425 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d801e960 con 0x7ff6dc136ac0 2024-04-19T01:44:47.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.709+0000 7ff6d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:47.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.709+0000 7ff6d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:44:47.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.709+0000 7ff6d2ffd640 10 monclient: dump: 2024-04-19T01:44:47.713 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:47.713 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:47.713 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.713 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.713 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:47.713 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:47.713 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:47.713 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:47.713 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:47.713 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:47.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.709+0000 7ff6d2ffd640 1 -- 172.21.15.62:0/203564425 <== mon.2 v2:172.21.15.62:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4717 (secure 0 0 0) 0x7ff6d8013420 con 0x7ff6dc136ac0 2024-04-19T01:44:47.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.709+0000 7ff6d2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:44:47.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.709+0000 7ff6d2ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:44:47.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.709+0000 7ff6d2ffd640 1 -- 172.21.15.62:0/203564425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6dc136ac0 msgr2=0x7ff6dc070780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:47.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.709+0000 7ff6d2ffd640 1 --2- 172.21.15.62:0/203564425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6dc136ac0 0x7ff6dc070780 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc139e60 tx=0x7ff6d8003770 comp rx=0 tx=0).stop 2024-04-19T01:44:47.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.709+0000 7ff6d2ffd640 1 -- 172.21.15.62:0/203564425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6dc136ac0 msgr2=0x7ff6dc070780 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:44:47.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.709+0000 7ff6d2ffd640 1 --2- 172.21.15.62:0/203564425 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff6dc136ac0 0x7ff6dc070780 secure :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc139e60 tx=0x7ff6d8003770 comp rx=0 tx=0).stop 2024-04-19T01:44:47.714 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-19T01:44:47.714 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "c", 2024-04-19T01:44:47.714 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 2, 2024-04-19T01:44:47.714 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "peon", 2024-04-19T01:44:47.714 INFO:teuthology.orchestra.run.smithi062.stdout: "election_epoch": 28, 2024-04-19T01:44:47.715 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 163914, 2024-04-19T01:44:47.715 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum": [ 2024-04-19T01:44:47.715 INFO:teuthology.orchestra.run.smithi062.stdout: 0, 2024-04-19T01:44:47.715 INFO:teuthology.orchestra.run.smithi062.stdout: 1, 2024-04-19T01:44:47.715 INFO:teuthology.orchestra.run.smithi062.stdout: 2 2024-04-19T01:44:47.715 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:44:47.715 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_age": 5, 2024-04-19T01:44:47.715 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:44:47.715 INFO:teuthology.orchestra.run.smithi062.stdout: "required_con": "2450521708007866372", 2024-04-19T01:44:47.715 INFO:teuthology.orchestra.run.smithi062.stdout: "required_mon": [ 2024-04-19T01:44:47.715 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:44:47.715 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:44:47.715 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:44:47.715 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:44:47.715 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:44:47.716 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:44:47.716 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:44:47.716 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:44:47.716 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:44:47.716 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:44:47.716 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:44:47.716 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:44:47.716 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:44:47.716 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_mon": [ 2024-04-19T01:44:47.716 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:44:47.716 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:44:47.716 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:44:47.716 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:44:47.716 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:44:47.716 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:44:47.717 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:44:47.717 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:44:47.717 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:44:47.717 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:44:47.717 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:44:47.717 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:47.717 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.717 INFO:teuthology.orchestra.run.smithi062.stdout: "outside_quorum": [], 2024-04-19T01:44:47.717 INFO:teuthology.orchestra.run.smithi062.stdout: "extra_probe_peers": [], 2024-04-19T01:44:47.717 INFO:teuthology.orchestra.run.smithi062.stdout: "sync_provider": [], 2024-04-19T01:44:47.717 INFO:teuthology.orchestra.run.smithi062.stdout: "monmap": { 2024-04-19T01:44:47.717 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 1, 2024-04-19T01:44:47.717 INFO:teuthology.orchestra.run.smithi062.stdout: "fsid": "6466e4b6-2520-485f-95b3-9401915fbbad", 2024-04-19T01:44:47.717 INFO:teuthology.orchestra.run.smithi062.stdout: "modified": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:44:47.717 INFO:teuthology.orchestra.run.smithi062.stdout: "created": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:44:47.718 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release": 19, 2024-04-19T01:44:47.718 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release_name": "squid", 2024-04-19T01:44:47.718 INFO:teuthology.orchestra.run.smithi062.stdout: "election_strategy": 1, 2024-04-19T01:44:47.718 INFO:teuthology.orchestra.run.smithi062.stdout: "disallowed_leaders: ": "", 2024-04-19T01:44:47.718 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false, 2024-04-19T01:44:47.718 INFO:teuthology.orchestra.run.smithi062.stdout: "tiebreaker_mon": "", 2024-04-19T01:44:47.718 INFO:teuthology.orchestra.run.smithi062.stdout: "removed_ranks: ": "", 2024-04-19T01:44:47.718 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:44:47.718 INFO:teuthology.orchestra.run.smithi062.stdout: "persistent": [ 2024-04-19T01:44:47.718 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:44:47.718 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:44:47.718 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:44:47.718 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:44:47.718 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:44:47.718 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:44:47.719 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:44:47.719 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:44:47.719 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:44:47.719 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:44:47.719 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:44:47.719 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:44:47.719 INFO:teuthology.orchestra.run.smithi062.stdout: "optional": [] 2024-04-19T01:44:47.719 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.719 INFO:teuthology.orchestra.run.smithi062.stdout: "mons": [ 2024-04-19T01:44:47.719 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.719 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 0, 2024-04-19T01:44:47.719 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "a", 2024-04-19T01:44:47.719 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:44:47.719 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:44:47.720 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.720 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:44:47.720 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3300", 2024-04-19T01:44:47.720 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:47.720 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.720 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.720 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:44:47.720 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789", 2024-04-19T01:44:47.720 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:47.720 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:47.720 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:47.720 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.720 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789/0", 2024-04-19T01:44:47.720 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6789/0", 2024-04-19T01:44:47.720 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:44:47.721 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:44:47.721 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:44:47.721 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.721 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.721 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 1, 2024-04-19T01:44:47.721 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "b", 2024-04-19T01:44:47.721 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:44:47.721 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:44:47.721 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.721 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:44:47.721 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:3300", 2024-04-19T01:44:47.721 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:47.721 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.721 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.722 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:44:47.722 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789", 2024-04-19T01:44:47.722 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:47.722 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:47.722 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:47.722 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.722 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789/0", 2024-04-19T01:44:47.722 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.71:6789/0", 2024-04-19T01:44:47.722 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:44:47.722 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:44:47.722 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:44:47.722 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.722 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.722 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 2, 2024-04-19T01:44:47.723 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "c", 2024-04-19T01:44:47.723 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:44:47.723 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:44:47.723 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.723 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:44:47.723 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3301", 2024-04-19T01:44:47.723 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:47.723 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.723 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.723 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:44:47.723 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790", 2024-04-19T01:44:47.723 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:47.723 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:47.723 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:47.724 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.724 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790/0", 2024-04-19T01:44:47.724 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6790/0", 2024-04-19T01:44:47.724 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:44:47.724 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:44:47.724 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:44:47.724 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:47.724 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:47.724 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.724 INFO:teuthology.orchestra.run.smithi062.stdout: "feature_map": { 2024-04-19T01:44:47.724 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": [ 2024-04-19T01:44:47.724 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.724 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:44:47.724 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:44:47.724 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:44:47.725 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:47.725 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:44:47.725 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": [ 2024-04-19T01:44:47.725 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.725 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:44:47.725 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:44:47.725 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:44:47.725 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:47.725 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:44:47.725 INFO:teuthology.orchestra.run.smithi062.stdout: "client": [ 2024-04-19T01:44:47.725 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.725 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:44:47.725 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:44:47.725 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:44:47.726 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:47.726 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:47.726 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.726 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false 2024-04-19T01:44:47.726 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-19T01:44:47.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.709+0000 7ff6e44fb640 1 -- 172.21.15.62:0/203564425 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7ff6c0036450 msgr2=0x7ff6c0038910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:47.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.709+0000 7ff6e44fb640 1 --2- 172.21.15.62:0/203564425 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7ff6c0036450 0x7ff6c0038910 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc13a0b0 tx=0x7ff6d4009820 comp rx=0 tx=0).stop 2024-04-19T01:44:47.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.709+0000 7ff6e44fb640 10 monclient: shutdown 2024-04-19T01:44:47.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.709+0000 7ff6e44fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:47.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.709+0000 7ff6e44fb640 1 -- 172.21.15.62:0/203564425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6dc0a47e0 msgr2=0x7ff6dc1cd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:47.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.709+0000 7ff6e44fb640 1 --2- 172.21.15.62:0/203564425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6dc0a47e0 0x7ff6dc1cd3e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc01cb70 tx=0x7ff6cc01ab10 comp rx=0 tx=0).stop 2024-04-19T01:44:47.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.709+0000 7ff6e2a98640 1 -- 172.21.15.62:0/203564425 reap_dead start 2024-04-19T01:44:47.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.709+0000 7ff6d2ffd640 1 -- 172.21.15.62:0/203564425 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 4738+0+0 (unknown 0 0 0) 0x7ff6cc0e34d0 con 0x7ff6dc0a47e0 2024-04-19T01:44:47.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.709+0000 7ff6e44fb640 1 -- 172.21.15.62:0/203564425 shutdown_connections 2024-04-19T01:44:47.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.709+0000 7ff6e44fb640 1 --2- 172.21.15.62:0/203564425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6dc0a47e0 0x7ff6dc1cd3e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:47.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.709+0000 7ff6e44fb640 1 -- 172.21.15.62:0/203564425 >> 172.21.15.62:0/203564425 conn(0x7ff6dc0a23d0 msgr2=0x7ff6dc06fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:47.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.709+0000 7ff6e44fb640 1 -- 172.21.15.62:0/203564425 shutdown_connections 2024-04-19T01:44:47.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.709+0000 7ff6e44fb640 1 -- 172.21.15.62:0/203564425 wait complete. 2024-04-19T01:44:47.728 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-19T01:44:47.728 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-19T01:44:47.729 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T01:44:47.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e2f96640 10 monclient: get_monmap_and_config 2024-04-19T01:44:47.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e2f96640 10 monclient: build_initial_monmap 2024-04-19T01:44:47.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e2f96640 10 monclient: monmap: 2024-04-19T01:44:47.845 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:47.846 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:47.846 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:44:47.850687+0000 2024-04-19T01:44:47.846 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:44:47.850687+0000 2024-04-19T01:44:47.846 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:47.846 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:47.846 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:44:47.846 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:44:47.846 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:44:47.846 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:44:47.846 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:44:47.846 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:47.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e2f96640 1 Processor -- start 2024-04-19T01:44:47.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e2f96640 1 -- start start 2024-04-19T01:44:47.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e2f96640 10 monclient: init 2024-04-19T01:44:47.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e2f96640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:47.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e2f96640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-04-19T01:44:47.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e2f96640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f44dc135670 0x7f44dc072a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:47.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e2f96640 10 monclient(hunting): picked mon.noname-c con 0x7f44dc135670 addr v2:172.21.15.71:3300/0 2024-04-19T01:44:47.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e2f96640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f44dc135a50 0x7f44dc0a34e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:47.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e2f96640 10 monclient(hunting): picked mon.noname-a con 0x7f44dc135a50 addr v2:172.21.15.62:3300/0 2024-04-19T01:44:47.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e2f96640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f44dc135eb0 0x7f44dc0a3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:47.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e2f96640 10 monclient(hunting): picked mon.noname-b con 0x7f44dc135eb0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:44:47.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e2f96640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:47.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e2f96640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f44dc0700c0 con 0x7f44dc135a50 2024-04-19T01:44:47.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e2f96640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:47.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e2f96640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f44dc072f70 con 0x7f44dc135670 2024-04-19T01:44:47.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e2f96640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:47.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e2f96640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f44dc070500 con 0x7f44dc135eb0 2024-04-19T01:44:47.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e2f96640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:47.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e2f96640 10 monclient(hunting): authenticate will time out at 1073.670230s 2024-04-19T01:44:47.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44dbfff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f44dc135a50 0x7f44dc0a34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:47.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e1533640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f44dc135eb0 0x7f44dc0a3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:47.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44dbfff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f44dc135a50 0x7f44dc0a34e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54254/0 (socket says 172.21.15.62:54254) 2024-04-19T01:44:47.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44dbfff640 1 -- 172.21.15.62:0/2865006159 learned_addr learned my addr 172.21.15.62:0/2865006159 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:44:47.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44dbfff640 10 monclient(hunting): get_auth_request con 0x7f44dc135a50 auth_method 0 2024-04-19T01:44:47.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:47.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44dbfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:47.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:47.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e1533640 10 monclient(hunting): get_auth_request con 0x7f44dc135eb0 auth_method 0 2024-04-19T01:44:47.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e1533640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:47.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e1533640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:47.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e1533640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:47.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:47.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:47.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:47.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e0d32640 1 --2- 172.21.15.62:0/2865006159 >> v2:172.21.15.71:3300/0 conn(0x7f44dc135670 0x7f44dc072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:47.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e1533640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:47.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e1533640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:47.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e1533640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:47.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e0d32640 10 monclient(hunting): get_auth_request con 0x7f44dc135670 auth_method 0 2024-04-19T01:44:47.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e0d32640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:47.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e0d32640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:47.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e0d32640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:47.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e1533640 10 monclient(hunting): handle_auth_done global_id 4808 payload 274 2024-04-19T01:44:47.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e1533640 1 -- 172.21.15.62:0/2865006159 >> v2:172.21.15.71:3300/0 conn(0x7f44dc135670 msgr2=0x7f44dc072a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:44:47.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e1533640 1 --2- 172.21.15.62:0/2865006159 >> v2:172.21.15.71:3300/0 conn(0x7f44dc135670 0x7f44dc072a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:47.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e1533640 1 -- 172.21.15.62:0/2865006159 >> v2:172.21.15.62:3300/0 conn(0x7f44dc135a50 msgr2=0x7f44dc0a34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:44:47.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e1533640 1 --2- 172.21.15.62:0/2865006159 >> v2:172.21.15.62:3300/0 conn(0x7f44dc135a50 0x7f44dc0a34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:47.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e1533640 10 monclient: _finish_hunting 0 2024-04-19T01:44:47.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e1533640 1 monclient: found mon.noname-b 2024-04-19T01:44:47.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e1533640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:47.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e1533640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:44:47.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.849+0000 7f44e1533640 1 -- 172.21.15.62:0/2865006159 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44dc0717b0 con 0x7f44dc135eb0 2024-04-19T01:44:47.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.853+0000 7f44e1533640 1 -- 172.21.15.62:0/2865006159 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f44dc135eb0 msgr2=0x7f44dc0a3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:47.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.853+0000 7f44e1533640 1 --2- 172.21.15.62:0/2865006159 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f44dc135eb0 0x7f44dc0a3a20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f44dc0704d0 tx=0x7f44d4020bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76b5b5c7cc8da5d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:47.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.853+0000 7f44db7fe640 1 -- 172.21.15.62:0/2865006159 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d400d0d0 con 0x7f44dc135eb0 2024-04-19T01:44:47.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.853+0000 7f44db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:47.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.853+0000 7f44db7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:44:47.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.853+0000 7f44db7fe640 10 monclient: dump: 2024-04-19T01:44:47.860 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:47.860 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:47.860 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.860 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.860 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:47.860 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:47.860 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:47.860 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:47.860 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:47.860 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:47.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.853+0000 7f44db7fe640 10 monclient: _finish_auth 0 2024-04-19T01:44:47.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.853+0000 7f44db7fe640 10 monclient: _check_auth_tickets 2024-04-19T01:44:47.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.853+0000 7f44db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:47.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.853+0000 7f44db7fe640 1 -- 172.21.15.62:0/2865006159 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44d400d270 con 0x7f44dc135eb0 2024-04-19T01:44:47.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.853+0000 7f44e2f96640 5 monclient: authenticate success, global_id 4808 2024-04-19T01:44:47.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.853+0000 7f44e2f96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:44:47.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.853+0000 7f44db7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:47.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.853+0000 7f44e2f96640 10 monclient: get_monmap_and_config success 2024-04-19T01:44:47.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.853+0000 7f44db7fe640 1 -- 172.21.15.62:0/2865006159 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d40a9880 con 0x7f44dc135eb0 2024-04-19T01:44:47.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.853+0000 7f44e2f96640 4 set_mon_vals no callback set 2024-04-19T01:44:47.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.853+0000 7f44db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:47.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.853+0000 7f44db7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:44:47.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.853+0000 7f44db7fe640 10 monclient: dump: 2024-04-19T01:44:47.861 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:47.861 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:47.861 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.861 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.862 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:47.862 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:47.862 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:47.862 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:47.862 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:47.862 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:47.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.853+0000 7f44e2f96640 10 monclient: shutdown 2024-04-19T01:44:47.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.853+0000 7f44e2f96640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:47.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.853+0000 7f44e2f96640 1 -- 172.21.15.62:0/2865006159 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f44dc135eb0 msgr2=0x7f44dc0a3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:47.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.853+0000 7f44e2f96640 1 --2- 172.21.15.62:0/2865006159 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f44dc135eb0 0x7f44dc0a3a20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f44dc0704d0 tx=0x7f44d4020bf0 comp rx=0 tx=0).stop 2024-04-19T01:44:47.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.853+0000 7f44e2f96640 1 -- 172.21.15.62:0/2865006159 shutdown_connections 2024-04-19T01:44:47.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.853+0000 7f44e2f96640 1 --2- 172.21.15.62:0/2865006159 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f44dc135eb0 0x7f44dc0a3a20 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:47.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.853+0000 7f44e2f96640 1 --2- 172.21.15.62:0/2865006159 >> v2:172.21.15.62:3300/0 conn(0x7f44dc135a50 0x7f44dc0a34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:47.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.853+0000 7f44e2f96640 1 --2- 172.21.15.62:0/2865006159 >> v2:172.21.15.71:3300/0 conn(0x7f44dc135670 0x7f44dc072a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:47.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.853+0000 7f44e2f96640 1 -- 172.21.15.62:0/2865006159 >> 172.21.15.62:0/2865006159 conn(0x7f44dc0a8100 msgr2=0x7f44dc0a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:47.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.853+0000 7f44e2f96640 1 -- 172.21.15.62:0/2865006159 shutdown_connections 2024-04-19T01:44:47.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e2f96640 1 -- 172.21.15.62:0/2865006159 wait complete. 2024-04-19T01:44:47.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e2f96640 10 monclient: build_initial_monmap 2024-04-19T01:44:47.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e2f96640 10 monclient: monmap: 2024-04-19T01:44:47.863 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:47.863 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:44:47.863 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:44:47.863 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:44:47.863 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:47.863 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:47.863 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:44:47.863 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:44:47.863 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:44:47.864 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:47.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e2f96640 1 Processor -- start 2024-04-19T01:44:47.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e2f96640 1 -- start start 2024-04-19T01:44:47.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e2f96640 10 monclient: init 2024-04-19T01:44:47.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e2f96640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:47.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e2f96640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-19T01:44:47.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e2f96640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44dc135670 0x7f44dc133b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:47.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e2f96640 10 monclient(hunting): picked mon.noname-b con 0x7f44dc135670 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:44:47.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e2f96640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44dc135a50 0x7f44dc134090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:47.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e2f96640 10 monclient(hunting): picked mon.noname-a con 0x7f44dc135a50 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:44:47.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e2f96640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f44dc139e00 0x7f44dc13a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:47.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e2f96640 10 monclient(hunting): picked mon.noname-c con 0x7f44dc139e00 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:44:47.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e2f96640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:47.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e2f96640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f44dc13d080 con 0x7f44dc135a50 2024-04-19T01:44:47.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e2f96640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:47.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e2f96640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f44dc13cf00 con 0x7f44dc135670 2024-04-19T01:44:47.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e2f96640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:47.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e2f96640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f44dc13d200 con 0x7f44dc139e00 2024-04-19T01:44:47.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e2f96640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:47.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e2f96640 10 monclient(hunting): authenticate will time out at 1073.677449s 2024-04-19T01:44:47.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44dbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44dc135a50 0x7f44dc134090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:47.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e1533640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f44dc139e00 0x7f44dc13a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:47.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44dbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44dc135a50 0x7f44dc134090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54264/0 (socket says 172.21.15.62:54264) 2024-04-19T01:44:47.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e1533640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f44dc139e00 0x7f44dc13a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:51572/0 (socket says 172.21.15.62:51572) 2024-04-19T01:44:47.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44dbfff640 1 -- 172.21.15.62:0/1371899380 learned_addr learned my addr 172.21.15.62:0/1371899380 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:44:47.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44dbfff640 10 monclient(hunting): get_auth_request con 0x7f44dc135a50 auth_method 0 2024-04-19T01:44:47.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:47.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44dbfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:47.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:47.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e1533640 10 monclient(hunting): get_auth_request con 0x7f44dc139e00 auth_method 0 2024-04-19T01:44:47.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e1533640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:47.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e1533640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:47.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e1533640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:47.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e1533640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:47.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e1533640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:47.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e1533640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:47.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:47.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:47.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:47.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e0d32640 1 --2- 172.21.15.62:0/1371899380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44dc135670 0x7f44dc133b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:47.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e0d32640 10 monclient(hunting): get_auth_request con 0x7f44dc135670 auth_method 0 2024-04-19T01:44:47.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e0d32640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:47.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e0d32640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:47.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e0d32640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:47.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e1533640 10 monclient(hunting): handle_auth_done global_id 4811 payload 824 2024-04-19T01:44:47.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e1533640 1 -- 172.21.15.62:0/1371899380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44dc135670 msgr2=0x7f44dc133b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:47.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e1533640 1 --2- 172.21.15.62:0/1371899380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44dc135670 0x7f44dc133b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:47.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e1533640 1 -- 172.21.15.62:0/1371899380 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44dc135a50 msgr2=0x7f44dc134090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:47.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e1533640 1 --2- 172.21.15.62:0/1371899380 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44dc135a50 0x7f44dc134090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:47.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e1533640 10 monclient: _finish_hunting 0 2024-04-19T01:44:47.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e1533640 1 monclient: found mon.noname-c 2024-04-19T01:44:47.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e1533640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:47.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e1533640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:44:47.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.857+0000 7f44e1533640 1 -- 172.21.15.62:0/1371899380 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44dc073210 con 0x7f44dc139e00 2024-04-19T01:44:47.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44e1533640 1 -- 172.21.15.62:0/1371899380 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f44dc139e00 msgr2=0x7f44dc13a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:47.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44e1533640 1 --2- 172.21.15.62:0/1371899380 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f44dc139e00 0x7f44dc13a2b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f44dc070680 tx=0x7f44d401d990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:47.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44d97fa640 1 -- 172.21.15.62:0/1371899380 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d40c0020 con 0x7f44dc139e00 2024-04-19T01:44:47.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:47.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44d97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:44:47.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44d97fa640 10 monclient: dump: 2024-04-19T01:44:47.868 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:47.868 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:47.868 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.868 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.868 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:47.868 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:47.868 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:47.868 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:47.868 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:47.868 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:47.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44d97fa640 10 monclient: _finish_auth 0 2024-04-19T01:44:47.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44d97fa640 10 monclient: _check_auth_tickets 2024-04-19T01:44:47.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44d97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:47.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44e2f96640 5 monclient: authenticate success, global_id 4811 2024-04-19T01:44:47.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44e2f96640 10 monclient: _renew_subs 2024-04-19T01:44:47.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44d97fa640 1 -- 172.21.15.62:0/1371899380 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44d400bb50 con 0x7f44dc139e00 2024-04-19T01:44:47.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44e2f96640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:44:47.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44e2f96640 1 -- 172.21.15.62:0/1371899380 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44dc0a1050 con 0x7f44dc139e00 2024-04-19T01:44:47.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44d97fa640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:47.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44d97fa640 1 -- 172.21.15.62:0/1371899380 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d400d0d0 con 0x7f44dc139e00 2024-04-19T01:44:47.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:47.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44d97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:44:47.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44daffd640 4 set_mon_vals no callback set 2024-04-19T01:44:47.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44d97fa640 10 monclient: dump: 2024-04-19T01:44:47.870 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:47.870 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:47.870 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.870 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.870 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:47.870 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:47.870 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:47.870 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:47.870 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:47.870 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:47.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44e2f96640 10 monclient: _renew_subs 2024-04-19T01:44:47.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44e2f96640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:44:47.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44e2f96640 1 -- 172.21.15.62:0/1371899380 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f44dc13beb0 con 0x7f44dc139e00 2024-04-19T01:44:47.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44d97fa640 1 -- 172.21.15.62:0/1371899380 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f44d40c3020 con 0x7f44dc139e00 2024-04-19T01:44:47.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44e2f96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:47.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44e2f96640 1 --2- 172.21.15.62:0/1371899380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44dc13a7f0 0x7f44dc13abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:47.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44e2f96640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:47.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44e2f96640 1 -- 172.21.15.62:0/1371899380 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f44dc0692b0 con 0x7f44dc13a7f0 2024-04-19T01:44:47.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44d97fa640 1 --2- 172.21.15.62:0/1371899380 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f44bc036420 0x7f44bc0388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:47.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44d97fa640 1 -- 172.21.15.62:0/1371899380 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(118..118 src has 1..118) ==== 4738+0+0 (secure 0 0 0) 0x7f44d40e85b0 con 0x7f44dc139e00 2024-04-19T01:44:47.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44e0d32640 1 --2- 172.21.15.62:0/1371899380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44dc13a7f0 0x7f44dc13abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:47.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44e0d32640 10 monclient: get_auth_request con 0x7f44dc13a7f0 auth_method 0 2024-04-19T01:44:47.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44e0d32640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:47.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44e0d32640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:47.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44e0d32640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:47.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44dbfff640 1 --2- 172.21.15.62:0/1371899380 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f44bc036420 0x7f44bc0388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:47.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44dbfff640 10 monclient: get_auth_request con 0x7f44bc036420 auth_method 0 2024-04-19T01:44:47.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44e0d32640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:47.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44e0d32640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:47.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44e0d32640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:47.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44e0d32640 10 monclient(hunting): handle_auth_done global_id 4759 payload 824 2024-04-19T01:44:47.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.861+0000 7f44e0d32640 1 -- 172.21.15.62:0/1371899380 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f44dc0008e0 con 0x7f44dc13a7f0 2024-04-19T01:44:47.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.865+0000 7f44dbfff640 1 --2- 172.21.15.62:0/1371899380 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f44bc036420 0x7f44bc0388e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f44dc13b740 tx=0x7f44c4009240 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:47.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.865+0000 7f44e0d32640 1 -- 172.21.15.62:0/1371899380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44dc13a7f0 msgr2=0x7f44dc13abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:47.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.865+0000 7f44e0d32640 1 --2- 172.21.15.62:0/1371899380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44dc13a7f0 0x7f44dc13abd0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f44cc00f4b0 tx=0x7f44cc00f970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:47.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.865+0000 7f44d97fa640 1 -- 172.21.15.62:0/1371899380 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44cc019020 con 0x7f44dc13a7f0 2024-04-19T01:44:47.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.865+0000 7f44d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:47.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.865+0000 7f44d97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:44:47.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.865+0000 7f44d97fa640 10 monclient: dump: 2024-04-19T01:44:47.875 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:47.875 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:47.875 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.875 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.875 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:47.875 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:47.875 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:47.875 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:47.875 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:47.875 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:47.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.865+0000 7f44d97fa640 1 -- 172.21.15.62:0/1371899380 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f44cc014a80 con 0x7f44dc13a7f0 2024-04-19T01:44:47.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.865+0000 7f44d97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:47.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.865+0000 7f44d97fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:44:47.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.865+0000 7f44d97fa640 1 -- 172.21.15.62:0/1371899380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44dc13a7f0 msgr2=0x7f44dc13abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:47.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.865+0000 7f44d97fa640 1 --2- 172.21.15.62:0/1371899380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44dc13a7f0 0x7f44dc13abd0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f44cc00f4b0 tx=0x7f44cc00f970 comp rx=0 tx=0).stop 2024-04-19T01:44:47.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.865+0000 7f44d97fa640 1 -- 172.21.15.62:0/1371899380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44dc13a7f0 msgr2=0x7f44dc13abd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:44:47.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.865+0000 7f44d97fa640 1 --2- 172.21.15.62:0/1371899380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44dc13a7f0 0x7f44dc13abd0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:47.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.873+0000 7f44e2f96640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:44:47.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.873+0000 7f44e2f96640 1 --2- 172.21.15.62:0/1371899380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44dc135eb0 0x7f44dc1362c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:47.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.873+0000 7f44e2f96640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:47.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.873+0000 7f44e2f96640 1 -- 172.21.15.62:0/1371899380 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f44dc13b110 con 0x7f44dc135eb0 2024-04-19T01:44:47.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.873+0000 7f44e0d32640 1 --2- 172.21.15.62:0/1371899380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44dc135eb0 0x7f44dc1362c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:47.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.873+0000 7f44e0d32640 10 monclient: get_auth_request con 0x7f44dc135eb0 auth_method 0 2024-04-19T01:44:47.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.873+0000 7f44e0d32640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:47.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.873+0000 7f44e0d32640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:47.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.873+0000 7f44e0d32640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:47.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.873+0000 7f44e0d32640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:47.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.873+0000 7f44e0d32640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:47.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.873+0000 7f44e0d32640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:47.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.873+0000 7f44e0d32640 10 monclient(hunting): handle_auth_done global_id 4762 payload 824 2024-04-19T01:44:47.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.873+0000 7f44e0d32640 1 -- 172.21.15.62:0/1371899380 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f44dc11e660 con 0x7f44dc135eb0 2024-04-19T01:44:47.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.873+0000 7f44e0d32640 1 -- 172.21.15.62:0/1371899380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44dc135eb0 msgr2=0x7f44dc1362c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:47.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.873+0000 7f44e0d32640 1 --2- 172.21.15.62:0/1371899380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44dc135eb0 0x7f44dc1362c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f44cc01a000 tx=0x7f44cc0046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:47.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.877+0000 7f44d97fa640 1 -- 172.21.15.62:0/1371899380 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44cc019030 con 0x7f44dc135eb0 2024-04-19T01:44:47.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.877+0000 7f44d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:47.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.877+0000 7f44d97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:44:47.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.877+0000 7f44d97fa640 10 monclient: dump: 2024-04-19T01:44:47.877 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:47.878 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:47.878 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.878 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:47.878 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:47.878 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:47.878 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:47.878 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:47.878 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:47.878 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:47.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.877+0000 7f44d97fa640 1 -- 172.21.15.62:0/1371899380 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f44cc020e40 con 0x7f44dc135eb0 2024-04-19T01:44:47.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.877+0000 7f44d97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:44:47.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.877+0000 7f44d97fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:44:47.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.877+0000 7f44d97fa640 1 -- 172.21.15.62:0/1371899380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44dc135eb0 msgr2=0x7f44dc1362c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:47.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.877+0000 7f44d97fa640 1 --2- 172.21.15.62:0/1371899380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44dc135eb0 0x7f44dc1362c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f44cc01a000 tx=0x7f44cc0046d0 comp rx=0 tx=0).stop 2024-04-19T01:44:47.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.877+0000 7f44d97fa640 1 -- 172.21.15.62:0/1371899380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44dc135eb0 msgr2=0x7f44dc1362c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:44:47.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.877+0000 7f44d97fa640 1 --2- 172.21.15.62:0/1371899380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44dc135eb0 0x7f44dc1362c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:47.879 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-19T01:44:47.879 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "b", 2024-04-19T01:44:47.879 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 1, 2024-04-19T01:44:47.879 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "peon", 2024-04-19T01:44:47.879 INFO:teuthology.orchestra.run.smithi062.stdout: "election_epoch": 28, 2024-04-19T01:44:47.879 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 164043, 2024-04-19T01:44:47.879 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum": [ 2024-04-19T01:44:47.879 INFO:teuthology.orchestra.run.smithi062.stdout: 0, 2024-04-19T01:44:47.880 INFO:teuthology.orchestra.run.smithi062.stdout: 1, 2024-04-19T01:44:47.880 INFO:teuthology.orchestra.run.smithi062.stdout: 2 2024-04-19T01:44:47.880 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:44:47.880 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_age": 5, 2024-04-19T01:44:47.880 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:44:47.880 INFO:teuthology.orchestra.run.smithi062.stdout: "required_con": "2450521708007866372", 2024-04-19T01:44:47.880 INFO:teuthology.orchestra.run.smithi062.stdout: "required_mon": [ 2024-04-19T01:44:47.880 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:44:47.880 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:44:47.880 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:44:47.880 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:44:47.880 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:44:47.880 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:44:47.880 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:44:47.881 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:44:47.881 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:44:47.881 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:44:47.881 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:44:47.881 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:44:47.881 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:44:47.881 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_mon": [ 2024-04-19T01:44:47.881 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:44:47.881 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:44:47.881 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:44:47.881 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:44:47.881 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:44:47.881 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:44:47.882 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:44:47.882 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:44:47.882 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:44:47.882 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:44:47.882 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:44:47.882 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:47.882 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.882 INFO:teuthology.orchestra.run.smithi062.stdout: "outside_quorum": [], 2024-04-19T01:44:47.882 INFO:teuthology.orchestra.run.smithi062.stdout: "extra_probe_peers": [], 2024-04-19T01:44:47.882 INFO:teuthology.orchestra.run.smithi062.stdout: "sync_provider": [], 2024-04-19T01:44:47.882 INFO:teuthology.orchestra.run.smithi062.stdout: "monmap": { 2024-04-19T01:44:47.882 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 1, 2024-04-19T01:44:47.882 INFO:teuthology.orchestra.run.smithi062.stdout: "fsid": "6466e4b6-2520-485f-95b3-9401915fbbad", 2024-04-19T01:44:47.882 INFO:teuthology.orchestra.run.smithi062.stdout: "modified": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:44:47.883 INFO:teuthology.orchestra.run.smithi062.stdout: "created": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:44:47.883 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release": 19, 2024-04-19T01:44:47.883 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release_name": "squid", 2024-04-19T01:44:47.883 INFO:teuthology.orchestra.run.smithi062.stdout: "election_strategy": 1, 2024-04-19T01:44:47.883 INFO:teuthology.orchestra.run.smithi062.stdout: "disallowed_leaders: ": "", 2024-04-19T01:44:47.883 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false, 2024-04-19T01:44:47.883 INFO:teuthology.orchestra.run.smithi062.stdout: "tiebreaker_mon": "", 2024-04-19T01:44:47.883 INFO:teuthology.orchestra.run.smithi062.stdout: "removed_ranks: ": "", 2024-04-19T01:44:47.883 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:44:47.883 INFO:teuthology.orchestra.run.smithi062.stdout: "persistent": [ 2024-04-19T01:44:47.883 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:44:47.883 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:44:47.883 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:44:47.883 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:44:47.884 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:44:47.884 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:44:47.884 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:44:47.884 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:44:47.884 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:44:47.884 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:44:47.884 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:44:47.884 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:44:47.884 INFO:teuthology.orchestra.run.smithi062.stdout: "optional": [] 2024-04-19T01:44:47.884 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.884 INFO:teuthology.orchestra.run.smithi062.stdout: "mons": [ 2024-04-19T01:44:47.884 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.884 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 0, 2024-04-19T01:44:47.885 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "a", 2024-04-19T01:44:47.885 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:44:47.885 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:44:47.885 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.885 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:44:47.885 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3300", 2024-04-19T01:44:47.885 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:47.885 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.885 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.885 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:44:47.885 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789", 2024-04-19T01:44:47.885 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:47.885 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:47.885 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:47.886 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.886 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789/0", 2024-04-19T01:44:47.886 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6789/0", 2024-04-19T01:44:47.886 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:44:47.886 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:44:47.886 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:44:47.886 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.886 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.886 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 1, 2024-04-19T01:44:47.886 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "b", 2024-04-19T01:44:47.886 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:44:47.886 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:44:47.887 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.887 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:44:47.887 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:3300", 2024-04-19T01:44:47.887 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:47.887 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.887 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.887 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:44:47.887 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789", 2024-04-19T01:44:47.887 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:47.887 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:47.887 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:47.887 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.887 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789/0", 2024-04-19T01:44:47.887 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.71:6789/0", 2024-04-19T01:44:47.887 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:44:47.888 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:44:47.888 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:44:47.888 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.888 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.888 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 2, 2024-04-19T01:44:47.888 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "c", 2024-04-19T01:44:47.888 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:44:47.888 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:44:47.888 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.888 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:44:47.888 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3301", 2024-04-19T01:44:47.888 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:47.888 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.888 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.888 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:44:47.889 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790", 2024-04-19T01:44:47.889 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:44:47.889 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:47.889 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:47.889 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.889 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790/0", 2024-04-19T01:44:47.889 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6790/0", 2024-04-19T01:44:47.889 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:44:47.889 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:44:47.889 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:44:47.889 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:47.889 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:47.889 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.889 INFO:teuthology.orchestra.run.smithi062.stdout: "feature_map": { 2024-04-19T01:44:47.889 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": [ 2024-04-19T01:44:47.890 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.890 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:44:47.890 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:44:47.890 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:44:47.890 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:47.890 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:44:47.890 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": [ 2024-04-19T01:44:47.890 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.890 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:44:47.890 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:44:47.890 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 5 2024-04-19T01:44:47.890 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:47.890 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:44:47.890 INFO:teuthology.orchestra.run.smithi062.stdout: "client": [ 2024-04-19T01:44:47.890 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.891 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:44:47.891 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:44:47.891 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 6 2024-04-19T01:44:47.891 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:47.891 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:44:47.891 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": [ 2024-04-19T01:44:47.891 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:44:47.891 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:44:47.891 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:44:47.891 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:44:47.891 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:44:47.891 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:44:47.891 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:44:47.891 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false 2024-04-19T01:44:47.892 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-19T01:44:47.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.877+0000 7f44e2f96640 1 -- 172.21.15.62:0/1371899380 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f44bc036420 msgr2=0x7f44bc0388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:47.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.877+0000 7f44e2f96640 1 --2- 172.21.15.62:0/1371899380 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f44bc036420 0x7f44bc0388e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f44dc13b740 tx=0x7f44c4009240 comp rx=0 tx=0).stop 2024-04-19T01:44:47.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.877+0000 7f44e2f96640 10 monclient: shutdown 2024-04-19T01:44:47.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.877+0000 7f44e2f96640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:47.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.877+0000 7f44e2f96640 1 -- 172.21.15.62:0/1371899380 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f44dc139e00 msgr2=0x7f44dc13a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:47.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.877+0000 7f44e2f96640 1 --2- 172.21.15.62:0/1371899380 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f44dc139e00 0x7f44dc13a2b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f44dc070680 tx=0x7f44d401d990 comp rx=0 tx=0).stop 2024-04-19T01:44:47.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.877+0000 7f44e1533640 1 -- 172.21.15.62:0/1371899380 reap_dead start 2024-04-19T01:44:47.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.877+0000 7f44e1533640 1 -- 172.21.15.62:0/1371899380 reap_dead start 2024-04-19T01:44:47.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.877+0000 7f44e2f96640 1 -- 172.21.15.62:0/1371899380 shutdown_connections 2024-04-19T01:44:47.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.877+0000 7f44e2f96640 1 -- 172.21.15.62:0/1371899380 >> 172.21.15.62:0/1371899380 conn(0x7f44dc0a8100 msgr2=0x7f44dc0a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:47.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.877+0000 7f44e2f96640 1 -- 172.21.15.62:0/1371899380 shutdown_connections 2024-04-19T01:44:47.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:47.877+0000 7f44e2f96640 1 -- 172.21.15.62:0/1371899380 wait complete. 2024-04-19T01:44:47.894 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-19T01:44:47.894 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-04-19T01:44:47.894 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-19T01:44:47.894 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-04-19T01:44:47.894 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-19T01:44:47.895 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-19T01:44:47.895 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:44:47.917 INFO:tasks.ceph.mon.b:Stopped 2024-04-19T01:44:47.918 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-19T01:44:47.918 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:44:48.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.029+0000 7f60089b3640 10 monclient: get_monmap_and_config 2024-04-19T01:44:48.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.029+0000 7f60089b3640 10 monclient: build_initial_monmap 2024-04-19T01:44:48.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.029+0000 7f60089b3640 10 monclient: monmap: 2024-04-19T01:44:48.027 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:44:48.032939+0000 2024-04-19T01:44:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:44:48.032939+0000 2024-04-19T01:44:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:44:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:44:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:44:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:44:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:44:48.028 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:48.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f60089b3640 1 Processor -- start 2024-04-19T01:44:48.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f60089b3640 1 -- start start 2024-04-19T01:44:48.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f60089b3640 10 monclient: init 2024-04-19T01:44:48.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f60089b3640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:48.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f60089b3640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-04-19T01:44:48.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f60089b3640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f60040a3410 0x7f60040a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:48.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f60089b3640 10 monclient(hunting): picked mon.noname-a con 0x7f60040a3410 addr v2:172.21.15.62:3300/0 2024-04-19T01:44:48.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f60089b3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60040a2900 0x7f60040a46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:48.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f60089b3640 10 monclient(hunting): picked mon.noname-b con 0x7f60040a2900 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:44:48.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f60089b3640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f60040a2d20 0x7f60040a4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:48.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f60089b3640 10 monclient(hunting): picked mon.noname-c con 0x7f60040a2d20 addr v2:172.21.15.71:3300/0 2024-04-19T01:44:48.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f60089b3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:48.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f60089b3640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f60040a1430 con 0x7f60040a3410 2024-04-19T01:44:48.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f60089b3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:48.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f60089b3640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f60040a1710 con 0x7f60040a2d20 2024-04-19T01:44:48.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f60089b3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:48.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f60089b3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f60040a19f0 con 0x7f60040a2900 2024-04-19T01:44:48.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f60089b3640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:48.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f60089b3640 10 monclient(hunting): authenticate will time out at 1073.852899s 2024-04-19T01:44:48.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f600259c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f60040a3410 0x7f60040a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:48.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f600259c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f60040a3410 0x7f60040a37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54302/0 (socket says 172.21.15.62:54302) 2024-04-19T01:44:48.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f600259c640 1 -- 172.21.15.62:0/3155226192 learned_addr learned my addr 172.21.15.62:0/3155226192 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:44:48.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f600259c640 10 monclient(hunting): get_auth_request con 0x7f60040a3410 auth_method 0 2024-04-19T01:44:48.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f600259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:48.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f600259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:48.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f600259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:48.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f5ffbfff640 1 --2- 172.21.15.62:0/3155226192 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60040a2900 0x7f60040a46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:48.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f6002d9d640 1 -- 172.21.15.62:0/3155226192 >> v2:172.21.15.71:3300/0 conn(0x7f60040a2d20 msgr2=0x7f60040a4c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-04-19T01:44:48.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f6002d9d640 1 --2- 172.21.15.62:0/3155226192 >> v2:172.21.15.71:3300/0 conn(0x7f60040a2d20 0x7f60040a4c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:44:48.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f5ffbfff640 10 monclient(hunting): get_auth_request con 0x7f60040a2900 auth_method 0 2024-04-19T01:44:48.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f5ffbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:48.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f5ffbfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:48.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f5ffbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:48.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f600259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:48.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f600259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:48.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f600259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:48.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f5ffbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:48.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f5ffbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:48.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f5ffbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:48.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f600259c640 10 monclient(hunting): handle_auth_done global_id 14217 payload 274 2024-04-19T01:44:48.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f600259c640 1 -- 172.21.15.62:0/3155226192 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60040a2900 msgr2=0x7f60040a46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:44:48.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f600259c640 1 --2- 172.21.15.62:0/3155226192 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60040a2900 0x7f60040a46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:48.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f600259c640 1 -- 172.21.15.62:0/3155226192 >> v2:172.21.15.71:3300/0 conn(0x7f60040a2d20 msgr2=0x7f60040a4c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:44:48.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f600259c640 1 --2- 172.21.15.62:0/3155226192 >> v2:172.21.15.71:3300/0 conn(0x7f60040a2d20 0x7f60040a4c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:48.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f600259c640 10 monclient: _finish_hunting 0 2024-04-19T01:44:48.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f600259c640 1 monclient: found mon.noname-a 2024-04-19T01:44:48.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f600259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:48.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f600259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:44:48.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.033+0000 7f600259c640 1 -- 172.21.15.62:0/3155226192 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60040a3d30 con 0x7f60040a3410 2024-04-19T01:44:48.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f600259c640 1 -- 172.21.15.62:0/3155226192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60040a3410 msgr2=0x7f60040a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:48.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f600259c640 1 --2- 172.21.15.62:0/3155226192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60040a3410 0x7f60040a37f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f60040a26b0 tx=0x7f5ff401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92980dfbaf1e1f1c server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:48.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f6001d9b640 1 -- 172.21.15.62:0/3155226192 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff40b1070 con 0x7f60040a3410 2024-04-19T01:44:48.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f6001d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:48.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f6001d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:44:48.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f6001d9b640 10 monclient: dump: 2024-04-19T01:44:48.039 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:48.039 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:48.039 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:48.039 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:48.039 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:48.040 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:48.040 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:48.040 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:48.040 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:48.040 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:48.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f6001d9b640 10 monclient: _finish_auth 0 2024-04-19T01:44:48.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f6001d9b640 10 monclient: _check_auth_tickets 2024-04-19T01:44:48.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f6001d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:48.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f6001d9b640 1 -- 172.21.15.62:0/3155226192 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ff40a8050 con 0x7f60040a3410 2024-04-19T01:44:48.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f60089b3640 5 monclient: authenticate success, global_id 14217 2024-04-19T01:44:48.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f60089b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:44:48.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f6001d9b640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:48.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f60089b3640 10 monclient: get_monmap_and_config success 2024-04-19T01:44:48.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f6001d9b640 1 -- 172.21.15.62:0/3155226192 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff40ac860 con 0x7f60040a3410 2024-04-19T01:44:48.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f6001d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:48.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f60089b3640 4 set_mon_vals no callback set 2024-04-19T01:44:48.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f6001d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:44:48.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f6001d9b640 10 monclient: dump: 2024-04-19T01:44:48.041 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:48.041 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:48.041 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:48.041 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:48.041 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:48.041 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:48.041 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:48.041 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:48.042 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:48.042 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:48.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f60089b3640 10 monclient: shutdown 2024-04-19T01:44:48.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f60089b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:48.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f60089b3640 1 -- 172.21.15.62:0/3155226192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60040a3410 msgr2=0x7f60040a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:48.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f60089b3640 1 --2- 172.21.15.62:0/3155226192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60040a3410 0x7f60040a37f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f60040a26b0 tx=0x7f5ff401c400 comp rx=0 tx=0).stop 2024-04-19T01:44:48.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f60089b3640 1 -- 172.21.15.62:0/3155226192 shutdown_connections 2024-04-19T01:44:48.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f60089b3640 1 --2- 172.21.15.62:0/3155226192 >> v2:172.21.15.71:3300/0 conn(0x7f60040a2d20 0x7f60040a4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:48.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f60089b3640 1 --2- 172.21.15.62:0/3155226192 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60040a2900 0x7f60040a46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:48.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f60089b3640 1 --2- 172.21.15.62:0/3155226192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60040a3410 0x7f60040a37f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:48.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f60089b3640 1 -- 172.21.15.62:0/3155226192 >> 172.21.15.62:0/3155226192 conn(0x7f60040aa2d0 msgr2=0x7f60040a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:48.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f60089b3640 1 -- 172.21.15.62:0/3155226192 shutdown_connections 2024-04-19T01:44:48.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f60089b3640 1 -- 172.21.15.62:0/3155226192 wait complete. 2024-04-19T01:44:48.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f60089b3640 10 monclient: build_initial_monmap 2024-04-19T01:44:48.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f60089b3640 10 monclient: monmap: 2024-04-19T01:44:48.043 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:48.043 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:44:48.043 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:44:48.043 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:44:48.043 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:48.043 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:48.043 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:44:48.043 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:44:48.043 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:44:48.043 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:48.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f60089b3640 1 Processor -- start 2024-04-19T01:44:48.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f60089b3640 1 -- start start 2024-04-19T01:44:48.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f60089b3640 10 monclient: init 2024-04-19T01:44:48.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f60089b3640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:48.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.037+0000 7f60089b3640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-19T01:44:48.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f60089b3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60040a2900 0x7f60041cf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:48.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f60089b3640 10 monclient(hunting): picked mon.noname-c con 0x7f60040a2900 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:44:48.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f60089b3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60040a2d20 0x7f600413b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:48.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f60089b3640 10 monclient(hunting): picked mon.noname-b con 0x7f60040a2d20 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:44:48.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f60089b3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60040a3410 0x7f6004123140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:48.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f60089b3640 10 monclient(hunting): picked mon.noname-a con 0x7f60040a3410 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:44:48.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f60089b3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:48.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f60089b3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6004141370 con 0x7f60040a3410 2024-04-19T01:44:48.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f60089b3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:48.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f60089b3640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f60041411f0 con 0x7f60040a2d20 2024-04-19T01:44:48.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f60089b3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:48.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f60089b3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f60041414f0 con 0x7f60040a2900 2024-04-19T01:44:48.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f60089b3640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:48.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f60089b3640 10 monclient(hunting): authenticate will time out at 1073.860066s 2024-04-19T01:44:48.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f600259c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60040a2900 0x7f60041cf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:48.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f600259c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60040a2900 0x7f60041cf8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:51634/0 (socket says 172.21.15.62:51634) 2024-04-19T01:44:48.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f600259c640 1 -- 172.21.15.62:0/1821741635 learned_addr learned my addr 172.21.15.62:0/1821741635 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:44:48.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f600259c640 10 monclient(hunting): get_auth_request con 0x7f60040a2900 auth_method 0 2024-04-19T01:44:48.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f600259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:48.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f600259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:48.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f600259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:48.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f5ffbfff640 1 -- 172.21.15.62:0/1821741635 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60040a2d20 msgr2=0x7f600413b6e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-04-19T01:44:48.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f5ffbfff640 1 --2- 172.21.15.62:0/1821741635 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60040a2d20 0x7f600413b6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:44:48.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f600259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:48.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f600259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:48.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f600259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:48.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f6002d9d640 1 --2- 172.21.15.62:0/1821741635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60040a3410 0x7f6004123140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:48.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f6002d9d640 10 monclient(hunting): get_auth_request con 0x7f60040a3410 auth_method 0 2024-04-19T01:44:48.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f6002d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:48.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f6002d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:48.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f6002d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:48.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f6002d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:48.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f6002d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:48.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f6002d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:48.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f600259c640 10 monclient(hunting): handle_auth_done global_id 4817 payload 824 2024-04-19T01:44:48.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f600259c640 1 -- 172.21.15.62:0/1821741635 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60040a2d20 msgr2=0x7f600413b6e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:44:48.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f600259c640 1 --2- 172.21.15.62:0/1821741635 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60040a2d20 0x7f600413b6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:48.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f600259c640 1 -- 172.21.15.62:0/1821741635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60040a3410 msgr2=0x7f6004123140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:48.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f600259c640 1 --2- 172.21.15.62:0/1821741635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60040a3410 0x7f6004123140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:48.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f600259c640 10 monclient: _finish_hunting 0 2024-04-19T01:44:48.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f600259c640 1 monclient: found mon.noname-c 2024-04-19T01:44:48.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f600259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:48.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f600259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:44:48.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f600259c640 1 -- 172.21.15.62:0/1821741635 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6004124d90 con 0x7f60040a2900 2024-04-19T01:44:48.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f600259c640 1 -- 172.21.15.62:0/1821741635 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60040a2900 msgr2=0x7f60041cf8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:48.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f600259c640 1 --2- 172.21.15.62:0/1821741635 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60040a2900 0x7f60041cf8b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5ff401cca0 tx=0x7f5ff401ac40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:48.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.041+0000 7f5ffaffd640 1 -- 172.21.15.62:0/1821741635 DispatchQueue::entry inject delay of 0.002000 2024-04-19T01:44:48.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.045+0000 7f5ffaffd640 1 -- 172.21.15.62:0/1821741635 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff40bb020 con 0x7f60040a2900 2024-04-19T01:44:48.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.045+0000 7f5ffaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:48.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.045+0000 7f5ffaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:44:48.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.045+0000 7f5ffaffd640 10 monclient: dump: 2024-04-19T01:44:48.049 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:48.049 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:48.050 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:48.050 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:48.050 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:48.050 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:48.050 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:48.050 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:48.050 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:48.050 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:48.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.045+0000 7f5ffaffd640 10 monclient: _finish_auth 0 2024-04-19T01:44:48.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.045+0000 7f5ffaffd640 10 monclient: _check_auth_tickets 2024-04-19T01:44:48.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.045+0000 7f5ffaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:48.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.045+0000 7f5ffaffd640 1 -- 172.21.15.62:0/1821741635 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ff40b7420 con 0x7f60040a2900 2024-04-19T01:44:48.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.045+0000 7f5ffaffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:48.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.045+0000 7f5ffaffd640 1 -- 172.21.15.62:0/1821741635 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff40b1040 con 0x7f60040a2900 2024-04-19T01:44:48.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.045+0000 7f5ffaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:48.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.045+0000 7f5ffaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:44:48.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.045+0000 7f5ffaffd640 10 monclient: dump: 2024-04-19T01:44:48.051 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:48.051 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:48.051 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:48.051 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:48.051 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:48.051 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:48.051 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:48.051 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:48.051 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:48.051 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:48.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.045+0000 7f60089b3640 5 monclient: authenticate success, global_id 4817 2024-04-19T01:44:48.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.045+0000 7f60089b3640 10 monclient: _renew_subs 2024-04-19T01:44:48.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.045+0000 7f60089b3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:44:48.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.045+0000 7f60089b3640 1 -- 172.21.15.62:0/1821741635 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60041cfdf0 con 0x7f60040a2900 2024-04-19T01:44:48.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.045+0000 7f600159a640 4 set_mon_vals no callback set 2024-04-19T01:44:48.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.045+0000 7f60089b3640 10 monclient: _renew_subs 2024-04-19T01:44:48.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.045+0000 7f60089b3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:44:48.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.045+0000 7f60089b3640 1 -- 172.21.15.62:0/1821741635 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f60041267e0 con 0x7f60040a2900 2024-04-19T01:44:48.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.045+0000 7f60089b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:48.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.045+0000 7f60089b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:48.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.045+0000 7f60089b3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:44:48.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.045+0000 7f5ffaffd640 1 -- 172.21.15.62:0/1821741635 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f5ff40be020 con 0x7f60040a2900 2024-04-19T01:44:48.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.045+0000 7f60089b3640 1 -- 172.21.15.62:0/1821741635 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60040692b0 con 0x7f60040a2900 2024-04-19T01:44:48.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.045+0000 7f5ffaffd640 1 --2- 172.21.15.62:0/1821741635 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5fe4036420 0x7f5fe40388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:48.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.045+0000 7f5ffaffd640 1 -- 172.21.15.62:0/1821741635 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(118..118 src has 1..118) ==== 4738+0+0 (secure 0 0 0) 0x7f5ff40e3680 con 0x7f60040a2900 2024-04-19T01:44:48.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.045+0000 7f5ffbfff640 1 --2- 172.21.15.62:0/1821741635 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5fe4036420 0x7f5fe40388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:48.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.045+0000 7f5ffbfff640 10 monclient: get_auth_request con 0x7f5fe4036420 auth_method 0 2024-04-19T01:44:48.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.049+0000 7f5ffbfff640 1 --2- 172.21.15.62:0/1821741635 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5fe4036420 0x7f5fe40388e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6004125880 tx=0x7f5fec009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:48.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.049+0000 7f5ffaffd640 1 -- 172.21.15.62:0/1821741635 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5ff40c5030 con 0x7f60040a2900 2024-04-19T01:44:48.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.049+0000 7f5ffaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:48.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.049+0000 7f5ffaffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:44:48.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.277+0000 7f60089b3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:44:48.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.277+0000 7f60089b3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:44:48.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.277+0000 7f60089b3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:44:48.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.277+0000 7f60089b3640 1 -- 172.21.15.62:0/1821741635 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6004126b20 con 0x7f60040a2900 2024-04-19T01:44:48.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.277+0000 7f5ffaffd640 1 -- 172.21.15.62:0/1821741635 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f5ff4016110 con 0x7f60040a2900 2024-04-19T01:44:48.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.277+0000 7f5ffaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:44:48.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.277+0000 7f5ffaffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:44:48.275 INFO:teuthology.orchestra.run.smithi062.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","modified":"2024-04-19T01:41:38.497795Z","created":"2024-04-19T01:41:38.497795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:44:48.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.281+0000 7f60089b3640 1 -- 172.21.15.62:0/1821741635 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5fe4036420 msgr2=0x7f5fe40388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:48.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.281+0000 7f60089b3640 1 --2- 172.21.15.62:0/1821741635 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5fe4036420 0x7f5fe40388e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6004125880 tx=0x7f5fec009190 comp rx=0 tx=0).stop 2024-04-19T01:44:48.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.281+0000 7f60089b3640 10 monclient: shutdown 2024-04-19T01:44:48.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.281+0000 7f60089b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:48.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.281+0000 7f60089b3640 1 -- 172.21.15.62:0/1821741635 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60040a2900 msgr2=0x7f60041cf8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:48.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.281+0000 7f60089b3640 1 --2- 172.21.15.62:0/1821741635 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60040a2900 0x7f60041cf8b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5ff401cca0 tx=0x7f5ff401ac40 comp rx=0 tx=0).stop 2024-04-19T01:44:48.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.281+0000 7f60089b3640 1 -- 172.21.15.62:0/1821741635 shutdown_connections 2024-04-19T01:44:48.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.281+0000 7f60089b3640 1 --2- 172.21.15.62:0/1821741635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60040a3410 0x7f6004123140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:48.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.281+0000 7f60089b3640 1 --2- 172.21.15.62:0/1821741635 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60040a2d20 0x7f600413b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:48.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.281+0000 7f60089b3640 1 --2- 172.21.15.62:0/1821741635 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5fe4036420 0x7f5fe40388e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:48.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.281+0000 7f60089b3640 1 --2- 172.21.15.62:0/1821741635 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f60040a2900 0x7f60041cf8b0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:48.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.281+0000 7f60089b3640 1 -- 172.21.15.62:0/1821741635 >> 172.21.15.62:0/1821741635 conn(0x7f60040aa2d0 msgr2=0x7f60040a7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:48.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.281+0000 7f60089b3640 1 -- 172.21.15.62:0/1821741635 shutdown_connections 2024-04-19T01:44:48.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:48.281+0000 7f60089b3640 1 -- 172.21.15.62:0/1821741635 wait complete. 2024-04-19T01:44:51.295 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:44:51.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d54b2a640 10 monclient: get_monmap_and_config 2024-04-19T01:44:51.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d54b2a640 10 monclient: build_initial_monmap 2024-04-19T01:44:51.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d54b2a640 10 monclient: monmap: 2024-04-19T01:44:51.430 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:51.430 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:51.430 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:44:51.435356+0000 2024-04-19T01:44:51.431 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:44:51.435356+0000 2024-04-19T01:44:51.431 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:51.431 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:51.431 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:44:51.431 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:44:51.431 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:44:51.431 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:44:51.431 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:44:51.431 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:51.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d54b2a640 1 Processor -- start 2024-04-19T01:44:51.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d54b2a640 1 -- start start 2024-04-19T01:44:51.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d54b2a640 10 monclient: init 2024-04-19T01:44:51.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d54b2a640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:51.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d54b2a640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-04-19T01:44:51.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d54b2a640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4d50137780 0x7f4d50137b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:51.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d54b2a640 10 monclient(hunting): picked mon.noname-c con 0x7f4d50137780 addr v2:172.21.15.71:3300/0 2024-04-19T01:44:51.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d54b2a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4d501380a0 addr v1:172.21.15.71:6789/0 2024-04-19T01:44:51.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d54b2a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d500a60b0 0x7f4d500a4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:51.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d54b2a640 10 monclient(hunting): picked mon.noname-b con 0x7f4d500a60b0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:44:51.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d54b2a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d5009c2e0 con 0x7f4d501380a0 2024-04-19T01:44:51.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d54b2a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:51.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d54b2a640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f4d5009bd20 con 0x7f4d50137780 2024-04-19T01:44:51.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d54b2a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:51.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d54b2a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f4d5009c000 con 0x7f4d500a60b0 2024-04-19T01:44:51.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d54b2a640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:51.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d54b2a640 10 monclient(hunting): authenticate will time out at 1077.255132s 2024-04-19T01:44:51.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d4ed9d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d500a60b0 0x7f4d500a4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:51.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d4e59c640 1 -- >> v2:172.21.15.71:3300/0 conn(0x7f4d50137780 msgr2=0x7f4d50137b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-04-19T01:44:51.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d4dd9b640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7f4d501380a0 legacy=0x7f4d500a45d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-04-19T01:44:51.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d4e59c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4d50137780 0x7f4d50137b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:44:51.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d4ed9d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d500a60b0 0x7f4d500a4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:51642/0 (socket says 172.21.15.62:51642) 2024-04-19T01:44:51.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d4ed9d640 1 -- 172.21.15.62:0/1872740714 learned_addr learned my addr 172.21.15.62:0/1872740714 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:44:51.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d4ed9d640 10 monclient(hunting): get_auth_request con 0x7f4d500a60b0 auth_method 0 2024-04-19T01:44:51.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d4ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:51.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d4ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:51.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d4ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:51.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d4ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:51.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d4ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:51.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d4ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:51.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d4ed9d640 10 monclient(hunting): handle_auth_done global_id 4820 payload 274 2024-04-19T01:44:51.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d4ed9d640 1 -- 172.21.15.62:0/1872740714 >> v2:172.21.15.71:3300/0 conn(0x7f4d50137780 msgr2=0x7f4d50137b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:44:51.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d4ed9d640 1 --2- 172.21.15.62:0/1872740714 >> v2:172.21.15.71:3300/0 conn(0x7f4d50137780 0x7f4d50137b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:51.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d4ed9d640 1 -- 172.21.15.62:0/1872740714 >> v1:172.21.15.71:6789/0 conn(0x7f4d501380a0 legacy=0x7f4d500a45d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:44:51.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d4ed9d640 10 monclient: _finish_hunting 0 2024-04-19T01:44:51.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d4ed9d640 1 monclient: found mon.noname-b 2024-04-19T01:44:51.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d4ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:51.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d4ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:44:51.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.433+0000 7f4d4ed9d640 1 -- 172.21.15.62:0/1872740714 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d50073210 con 0x7f4d500a60b0 2024-04-19T01:44:51.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.437+0000 7f4d4ed9d640 1 -- 172.21.15.62:0/1872740714 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d500a60b0 msgr2=0x7f4d500a4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:51.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.437+0000 7f4d4ed9d640 1 --2- 172.21.15.62:0/1872740714 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d500a60b0 0x7f4d500a4ce0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4d40009870 tx=0x7f4d4001c260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80c5f9a207fa5e76 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:51.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4d59a640 1 -- 172.21.15.62:0/1872740714 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d400b0070 con 0x7f4d500a60b0 2024-04-19T01:44:51.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:51.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:44:51.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4d59a640 10 monclient: dump: 2024-04-19T01:44:51.443 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:51.443 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:51.443 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:51.443 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:51.444 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:51.444 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:51.444 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:51.444 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:51.444 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:51.444 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:51.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4d59a640 10 monclient: _finish_auth 0 2024-04-19T01:44:51.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4d59a640 10 monclient: _check_auth_tickets 2024-04-19T01:44:51.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:51.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 5 monclient: authenticate success, global_id 4820 2024-04-19T01:44:51.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:44:51.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4d59a640 1 -- 172.21.15.62:0/1872740714 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d400ab600 con 0x7f4d500a60b0 2024-04-19T01:44:51.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4d59a640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:51.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 10 monclient: get_monmap_and_config success 2024-04-19T01:44:51.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 4 set_mon_vals no callback set 2024-04-19T01:44:51.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4d59a640 1 -- 172.21.15.62:0/1872740714 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d400ab9d0 con 0x7f4d500a60b0 2024-04-19T01:44:51.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:51.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:44:51.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4d59a640 10 monclient: dump: 2024-04-19T01:44:51.445 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:51.445 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:51.445 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:51.445 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:51.445 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:51.445 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:51.445 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:51.445 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:51.445 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:51.445 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:51.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 10 monclient: shutdown 2024-04-19T01:44:51.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:51.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 1 -- 172.21.15.62:0/1872740714 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d500a60b0 msgr2=0x7f4d500a4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:51.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 1 --2- 172.21.15.62:0/1872740714 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d500a60b0 0x7f4d500a4ce0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4d40009870 tx=0x7f4d4001c260 comp rx=0 tx=0).stop 2024-04-19T01:44:51.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 1 -- 172.21.15.62:0/1872740714 shutdown_connections 2024-04-19T01:44:51.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 1 --2- 172.21.15.62:0/1872740714 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d500a60b0 0x7f4d500a4ce0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:51.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 1 --2- 172.21.15.62:0/1872740714 >> v2:172.21.15.71:3300/0 conn(0x7f4d50137780 0x7f4d50137b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:51.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 1 -- 172.21.15.62:0/1872740714 >> 172.21.15.62:0/1872740714 conn(0x7f4d500aa1d0 msgr2=0x7f4d500a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:51.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 1 -- 172.21.15.62:0/1872740714 shutdown_connections 2024-04-19T01:44:51.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 1 -- 172.21.15.62:0/1872740714 wait complete. 2024-04-19T01:44:51.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 10 monclient: build_initial_monmap 2024-04-19T01:44:51.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 10 monclient: monmap: 2024-04-19T01:44:51.446 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:51.446 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:44:51.446 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:44:51.447 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:44:51.447 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:51.447 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:51.447 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:44:51.447 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:44:51.447 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:44:51.447 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:51.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 1 Processor -- start 2024-04-19T01:44:51.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 1 -- start start 2024-04-19T01:44:51.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 10 monclient: init 2024-04-19T01:44:51.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:51.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-19T01:44:51.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d500a60b0 0x7f4d501cf710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:51.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 10 monclient(hunting): picked mon.noname-c con 0x7f4d500a60b0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:44:51.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d50137780 0x7f4d50136670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:51.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 10 monclient(hunting): picked mon.noname-b con 0x7f4d50137780 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:44:51.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d501380a0 0x7f4d5013c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:51.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 10 monclient(hunting): picked mon.noname-a con 0x7f4d501380a0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:44:51.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:51.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4d50142170 con 0x7f4d501380a0 2024-04-19T01:44:51.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:51.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4d500a5430 con 0x7f4d50137780 2024-04-19T01:44:51.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:51.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f4d501422f0 con 0x7f4d500a60b0 2024-04-19T01:44:51.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:51.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d54b2a640 10 monclient(hunting): authenticate will time out at 1077.262399s 2024-04-19T01:44:51.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4e59c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d500a60b0 0x7f4d501cf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:51.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4ed9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d501380a0 0x7f4d5013c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:51.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4ed9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d501380a0 0x7f4d5013c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54330/0 (socket says 172.21.15.62:54330) 2024-04-19T01:44:51.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4ed9d640 1 -- 172.21.15.62:0/2267945984 learned_addr learned my addr 172.21.15.62:0/2267945984 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:44:51.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4ed9d640 10 monclient(hunting): get_auth_request con 0x7f4d501380a0 auth_method 0 2024-04-19T01:44:51.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:51.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:51.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:51.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4dd9b640 1 -- 172.21.15.62:0/2267945984 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d50137780 msgr2=0x7f4d50136670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-04-19T01:44:51.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4e59c640 10 monclient(hunting): get_auth_request con 0x7f4d500a60b0 auth_method 0 2024-04-19T01:44:51.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:51.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4e59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:51.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:51.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4dd9b640 1 --2- 172.21.15.62:0/2267945984 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d50137780 0x7f4d50136670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:44:51.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:51.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:51.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:51.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:51.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:51.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:51.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4ed9d640 10 monclient(hunting): handle_auth_done global_id 14223 payload 824 2024-04-19T01:44:51.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4ed9d640 1 -- 172.21.15.62:0/2267945984 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d500a60b0 msgr2=0x7f4d501cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:51.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4ed9d640 1 --2- 172.21.15.62:0/2267945984 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d500a60b0 0x7f4d501cf710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:51.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4ed9d640 1 -- 172.21.15.62:0/2267945984 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d50137780 msgr2=0x7f4d50136670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:44:51.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4ed9d640 1 --2- 172.21.15.62:0/2267945984 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d50137780 0x7f4d50136670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:51.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4ed9d640 10 monclient: _finish_hunting 0 2024-04-19T01:44:51.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4ed9d640 1 monclient: found mon.noname-a 2024-04-19T01:44:51.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:51.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:44:51.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4ed9d640 1 -- 172.21.15.62:0/2267945984 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d500a3420 con 0x7f4d501380a0 2024-04-19T01:44:51.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.441+0000 7f4d4e59c640 1 --2- 172.21.15.62:0/2267945984 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d500a60b0 0x7f4d501cf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:44:51.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.445+0000 7f4d4ed9d640 1 -- 172.21.15.62:0/2267945984 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d501380a0 msgr2=0x7f4d5013c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:51.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.445+0000 7f4d4ed9d640 1 --2- 172.21.15.62:0/2267945984 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d501380a0 0x7f4d5013c570 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4d4001cb00 tx=0x7f4d4001aaa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:51.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.445+0000 7f4d46ffd640 1 -- 172.21.15.62:0/2267945984 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d400ba020 con 0x7f4d501380a0 2024-04-19T01:44:51.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.445+0000 7f4d46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:51.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.445+0000 7f4d46ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:44:51.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.445+0000 7f4d46ffd640 10 monclient: dump: 2024-04-19T01:44:51.451 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:51.451 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:51.451 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:51.451 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:51.451 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:51.451 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:51.451 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:51.451 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:51.451 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:51.451 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:51.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.445+0000 7f4d46ffd640 10 monclient: _finish_auth 0 2024-04-19T01:44:51.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.445+0000 7f4d46ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:44:51.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.445+0000 7f4d46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:51.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.445+0000 7f4d54b2a640 5 monclient: authenticate success, global_id 14223 2024-04-19T01:44:51.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.445+0000 7f4d46ffd640 1 -- 172.21.15.62:0/2267945984 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d400b6630 con 0x7f4d501380a0 2024-04-19T01:44:51.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.445+0000 7f4d54b2a640 10 monclient: _renew_subs 2024-04-19T01:44:51.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.445+0000 7f4d54b2a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:44:51.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.445+0000 7f4d54b2a640 1 -- 172.21.15.62:0/2267945984 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d5013e1c0 con 0x7f4d501380a0 2024-04-19T01:44:51.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.445+0000 7f4d46ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:51.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.445+0000 7f4d46ffd640 1 -- 172.21.15.62:0/2267945984 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d400b0040 con 0x7f4d501380a0 2024-04-19T01:44:51.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.445+0000 7f4d46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:51.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.445+0000 7f4d46ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:44:51.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.445+0000 7f4d46ffd640 10 monclient: dump: 2024-04-19T01:44:51.452 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:51.452 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:51.452 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:51.453 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:51.453 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:51.453 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:51.453 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:51.453 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:51.453 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:51.453 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:51.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.445+0000 7f4d47fff640 4 set_mon_vals no callback set 2024-04-19T01:44:51.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.445+0000 7f4d54b2a640 10 monclient: _renew_subs 2024-04-19T01:44:51.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.445+0000 7f4d54b2a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:44:51.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.445+0000 7f4d54b2a640 1 -- 172.21.15.62:0/2267945984 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d5013eaa0 con 0x7f4d501380a0 2024-04-19T01:44:51.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.445+0000 7f4d54b2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:51.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.445+0000 7f4d54b2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:51.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.445+0000 7f4d54b2a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:44:51.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.445+0000 7f4d54b2a640 1 -- 172.21.15.62:0/2267945984 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d50068ab0 con 0x7f4d501380a0 2024-04-19T01:44:51.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.445+0000 7f4d46ffd640 1 -- 172.21.15.62:0/2267945984 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f4d400bd020 con 0x7f4d501380a0 2024-04-19T01:44:51.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.445+0000 7f4d46ffd640 1 --2- 172.21.15.62:0/2267945984 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4d30036420 0x7f4d300388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:51.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.445+0000 7f4d46ffd640 1 -- 172.21.15.62:0/2267945984 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 4738+0+0 (secure 0 0 0) 0x7f4d400e39d0 con 0x7f4d501380a0 2024-04-19T01:44:51.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.445+0000 7f4d4e59c640 1 --2- 172.21.15.62:0/2267945984 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4d30036420 0x7f4d300388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:51.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.445+0000 7f4d4e59c640 10 monclient: get_auth_request con 0x7f4d30036420 auth_method 0 2024-04-19T01:44:51.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.449+0000 7f4d4e59c640 1 --2- 172.21.15.62:0/2267945984 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4d30036420 0x7f4d300388e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4d48009420 tx=0x7f4d4800d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:51.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.449+0000 7f4d46ffd640 1 -- 172.21.15.62:0/2267945984 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4d400c4030 con 0x7f4d501380a0 2024-04-19T01:44:51.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.449+0000 7f4d46ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:51.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.449+0000 7f4d46ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:44:51.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.677+0000 7f4d54b2a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:44:51.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.677+0000 7f4d54b2a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:44:51.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.677+0000 7f4d54b2a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:44:51.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.677+0000 7f4d54b2a640 1 -- 172.21.15.62:0/2267945984 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4d500008e0 con 0x7f4d501380a0 2024-04-19T01:44:51.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.677+0000 7f4d46ffd640 1 -- 172.21.15.62:0/2267945984 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f4d5009c2e0 con 0x7f4d501380a0 2024-04-19T01:44:51.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.677+0000 7f4d46ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:44:51.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.677+0000 7f4d46ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:44:51.674 INFO:teuthology.orchestra.run.smithi062.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","modified":"2024-04-19T01:41:38.497795Z","created":"2024-04-19T01:41:38.497795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:44:51.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.681+0000 7f4d54b2a640 1 -- 172.21.15.62:0/2267945984 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4d30036420 msgr2=0x7f4d300388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:51.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.681+0000 7f4d54b2a640 1 --2- 172.21.15.62:0/2267945984 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4d30036420 0x7f4d300388e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4d48009420 tx=0x7f4d4800d040 comp rx=0 tx=0).stop 2024-04-19T01:44:51.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.681+0000 7f4d54b2a640 10 monclient: shutdown 2024-04-19T01:44:51.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.681+0000 7f4d54b2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:51.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.681+0000 7f4d54b2a640 1 -- 172.21.15.62:0/2267945984 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d501380a0 msgr2=0x7f4d5013c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:51.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.681+0000 7f4d54b2a640 1 --2- 172.21.15.62:0/2267945984 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d501380a0 0x7f4d5013c570 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4d4001cb00 tx=0x7f4d4001aaa0 comp rx=0 tx=0).stop 2024-04-19T01:44:51.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.681+0000 7f4d54b2a640 1 -- 172.21.15.62:0/2267945984 shutdown_connections 2024-04-19T01:44:51.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.681+0000 7f4d54b2a640 1 --2- 172.21.15.62:0/2267945984 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d501380a0 0x7f4d5013c570 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:51.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.681+0000 7f4d54b2a640 1 --2- 172.21.15.62:0/2267945984 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d50137780 0x7f4d50136670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:51.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.681+0000 7f4d54b2a640 1 --2- 172.21.15.62:0/2267945984 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4d30036420 0x7f4d300388e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:51.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.681+0000 7f4d54b2a640 1 --2- 172.21.15.62:0/2267945984 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d500a60b0 0x7f4d501cf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:51.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.681+0000 7f4d54b2a640 1 -- 172.21.15.62:0/2267945984 >> 172.21.15.62:0/2267945984 conn(0x7f4d500aa1d0 msgr2=0x7f4d500a79d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:51.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.681+0000 7f4d54b2a640 1 -- 172.21.15.62:0/2267945984 shutdown_connections 2024-04-19T01:44:51.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:51.681+0000 7f4d54b2a640 1 -- 172.21.15.62:0/2267945984 wait complete. 2024-04-19T01:44:54.696 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:44:54.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f76387c3640 10 monclient: get_monmap_and_config 2024-04-19T01:44:54.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f76387c3640 10 monclient: build_initial_monmap 2024-04-19T01:44:54.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f76387c3640 10 monclient: monmap: 2024-04-19T01:44:54.830 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:54.830 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:54.830 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:44:54.835014+0000 2024-04-19T01:44:54.830 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:44:54.835014+0000 2024-04-19T01:44:54.830 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:54.830 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:54.830 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:44:54.830 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:44:54.830 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:44:54.830 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:44:54.831 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:44:54.831 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:54.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f76387c3640 1 Processor -- start 2024-04-19T01:44:54.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f76387c3640 1 -- start start 2024-04-19T01:44:54.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f76387c3640 10 monclient: init 2024-04-19T01:44:54.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f76387c3640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:54.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f76387c3640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-04-19T01:44:54.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f76387c3640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f76300a47e0 0x7f76300a4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:54.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f76387c3640 10 monclient(hunting): picked mon.noname-a con 0x7f76300a47e0 addr v2:172.21.15.62:3300/0 2024-04-19T01:44:54.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f76387c3640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f76300a5100 0x7f76301350b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:54.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f76387c3640 10 monclient(hunting): picked mon.noname-c con 0x7f76300a5100 addr v2:172.21.15.71:3300/0 2024-04-19T01:44:54.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f76387c3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f76301355f0 addr v1:172.21.15.71:6789/0 2024-04-19T01:44:54.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f76387c3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f763009f7c0 con 0x7f76301355f0 2024-04-19T01:44:54.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f76387c3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:54.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f76387c3640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f763009ee90 con 0x7f76300a47e0 2024-04-19T01:44:54.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f76387c3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:54.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f76387c3640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f763009c2d0 con 0x7f76300a5100 2024-04-19T01:44:54.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f76387c3640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:54.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f76387c3640 10 monclient(hunting): authenticate will time out at 1080.654516s 2024-04-19T01:44:54.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f7636d60640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7f76301355f0 legacy=0x7f763013c5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-04-19T01:44:54.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f7635d5e640 1 -- >> v2:172.21.15.71:3300/0 conn(0x7f76300a5100 msgr2=0x7f76301350b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-04-19T01:44:54.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f763655f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f76300a47e0 0x7f76300a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:54.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f7635d5e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f76300a5100 0x7f76301350b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:44:54.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f763655f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f76300a47e0 0x7f76300a4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54338/0 (socket says 172.21.15.62:54338) 2024-04-19T01:44:54.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f763655f640 1 -- 172.21.15.62:0/336025257 learned_addr learned my addr 172.21.15.62:0/336025257 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:44:54.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f763655f640 10 monclient(hunting): get_auth_request con 0x7f76300a47e0 auth_method 0 2024-04-19T01:44:54.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f763655f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:54.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f763655f640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:54.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f763655f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:54.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f763655f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:54.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f763655f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:54.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f763655f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:54.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f763655f640 10 monclient(hunting): handle_auth_done global_id 14226 payload 274 2024-04-19T01:44:54.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f763655f640 1 -- 172.21.15.62:0/336025257 >> v2:172.21.15.71:3300/0 conn(0x7f76300a5100 msgr2=0x7f76301350b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:44:54.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f763655f640 1 --2- 172.21.15.62:0/336025257 >> v2:172.21.15.71:3300/0 conn(0x7f76300a5100 0x7f76301350b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:54.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f763655f640 1 -- 172.21.15.62:0/336025257 >> v1:172.21.15.71:6789/0 conn(0x7f76301355f0 legacy=0x7f763013c5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:44:54.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f763655f640 10 monclient: _finish_hunting 0 2024-04-19T01:44:54.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f763655f640 1 monclient: found mon.noname-a 2024-04-19T01:44:54.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f763655f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:54.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f763655f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:44:54.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.833+0000 7f763655f640 1 -- 172.21.15.62:0/336025257 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f763009faa0 con 0x7f76300a47e0 2024-04-19T01:44:54.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.837+0000 7f763655f640 1 -- 172.21.15.62:0/336025257 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76300a47e0 msgr2=0x7f76300a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:54.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.837+0000 7f763655f640 1 --2- 172.21.15.62:0/336025257 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76300a47e0 0x7f76300a4bc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f762c009870 tx=0x7f762c01c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a3cf0de64e13109 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:54.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.837+0000 7f763555d640 1 -- 172.21.15.62:0/336025257 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f762c0b0070 con 0x7f76300a47e0 2024-04-19T01:44:54.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.837+0000 7f763555d640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:54.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.837+0000 7f763555d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:44:54.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.837+0000 7f763555d640 10 monclient: dump: 2024-04-19T01:44:54.840 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:54.840 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:54.840 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:54.840 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:54.841 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:54.841 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:54.841 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:54.841 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:54.841 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:54.841 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:54.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.837+0000 7f763555d640 10 monclient: _finish_auth 0 2024-04-19T01:44:54.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.837+0000 7f763555d640 10 monclient: _check_auth_tickets 2024-04-19T01:44:54.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.837+0000 7f763555d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:54.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.837+0000 7f763555d640 1 -- 172.21.15.62:0/336025257 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f762c0ab600 con 0x7f76300a47e0 2024-04-19T01:44:54.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.837+0000 7f76387c3640 5 monclient: authenticate success, global_id 14226 2024-04-19T01:44:54.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.837+0000 7f76387c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:44:54.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.837+0000 7f763555d640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:54.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.837+0000 7f76387c3640 10 monclient: get_monmap_and_config success 2024-04-19T01:44:54.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.837+0000 7f763555d640 1 -- 172.21.15.62:0/336025257 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f762c0ab9d0 con 0x7f76300a47e0 2024-04-19T01:44:54.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.837+0000 7f763555d640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:54.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.837+0000 7f76387c3640 4 set_mon_vals no callback set 2024-04-19T01:44:54.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.837+0000 7f763555d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:44:54.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.837+0000 7f763555d640 10 monclient: dump: 2024-04-19T01:44:54.842 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:54.842 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:54.842 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:54.842 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:54.842 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:54.842 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:54.842 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:54.842 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:54.842 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:54.843 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:54.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f76387c3640 10 monclient: shutdown 2024-04-19T01:44:54.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f76387c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:54.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f76387c3640 1 -- 172.21.15.62:0/336025257 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76300a47e0 msgr2=0x7f76300a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:54.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f76387c3640 1 --2- 172.21.15.62:0/336025257 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76300a47e0 0x7f76300a4bc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f762c009870 tx=0x7f762c01c340 comp rx=0 tx=0).stop 2024-04-19T01:44:54.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f76387c3640 1 -- 172.21.15.62:0/336025257 shutdown_connections 2024-04-19T01:44:54.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f76387c3640 1 --2- 172.21.15.62:0/336025257 >> v2:172.21.15.71:3300/0 conn(0x7f76300a5100 0x7f76301350b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:54.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f76387c3640 1 --2- 172.21.15.62:0/336025257 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76300a47e0 0x7f76300a4bc0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:54.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f76387c3640 1 -- 172.21.15.62:0/336025257 >> 172.21.15.62:0/336025257 conn(0x7f76300a23d0 msgr2=0x7f76300a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:54.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f76387c3640 1 -- 172.21.15.62:0/336025257 shutdown_connections 2024-04-19T01:44:54.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f76387c3640 1 -- 172.21.15.62:0/336025257 wait complete. 2024-04-19T01:44:54.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f76387c3640 10 monclient: build_initial_monmap 2024-04-19T01:44:54.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f76387c3640 10 monclient: monmap: 2024-04-19T01:44:54.844 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:54.844 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:44:54.844 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:44:54.844 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:44:54.844 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:54.844 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:54.844 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:44:54.844 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:44:54.844 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:44:54.844 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:54.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f76387c3640 1 Processor -- start 2024-04-19T01:44:54.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f76387c3640 1 -- start start 2024-04-19T01:44:54.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f76387c3640 10 monclient: init 2024-04-19T01:44:54.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f76387c3640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:54.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f76387c3640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-19T01:44:54.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f76387c3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76300a47e0 0x7f7630131250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:54.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f76387c3640 10 monclient(hunting): picked mon.noname-b con 0x7f76300a47e0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:44:54.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f76387c3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76300a5100 0x7f7630139040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:54.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f76387c3640 10 monclient(hunting): picked mon.noname-a con 0x7f76300a5100 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:44:54.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f76387c3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f76301355f0 0x7f7630139580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:54.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f76387c3640 10 monclient(hunting): picked mon.noname-c con 0x7f76301355f0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:44:54.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f76387c3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:54.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f76387c3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f763009f1d0 con 0x7f76300a5100 2024-04-19T01:44:54.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f76387c3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:54.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f76387c3640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f763009f050 con 0x7f76300a47e0 2024-04-19T01:44:54.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f76387c3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:54.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f76387c3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f763009c760 con 0x7f76301355f0 2024-04-19T01:44:54.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f76387c3640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:54.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f76387c3640 10 monclient(hunting): authenticate will time out at 1080.661736s 2024-04-19T01:44:54.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f763655f640 1 -- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76300a47e0 msgr2=0x7f7630131250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-04-19T01:44:54.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f763655f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76300a47e0 0x7f7630131250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:44:54.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f7635d5e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76300a5100 0x7f7630139040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:54.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f7636d60640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f76301355f0 0x7f7630139580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:54.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f7635d5e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76300a5100 0x7f7630139040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54344/0 (socket says 172.21.15.62:54344) 2024-04-19T01:44:54.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f7635d5e640 1 -- 172.21.15.62:0/2297685527 learned_addr learned my addr 172.21.15.62:0/2297685527 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:44:54.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f7635d5e640 10 monclient(hunting): get_auth_request con 0x7f76300a5100 auth_method 0 2024-04-19T01:44:54.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f7635d5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:54.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f7635d5e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:54.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f7635d5e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:54.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f7636d60640 10 monclient(hunting): get_auth_request con 0x7f76301355f0 auth_method 0 2024-04-19T01:44:54.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f7636d60640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:54.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f7636d60640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:54.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f7636d60640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:54.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f7635d5e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:54.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f7635d5e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:54.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f7635d5e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:54.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f7636d60640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:44:54.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f7636d60640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:44:54.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f7636d60640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:44:54.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f7636d60640 10 monclient(hunting): handle_auth_done global_id 4826 payload 824 2024-04-19T01:44:54.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f7636d60640 1 -- 172.21.15.62:0/2297685527 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76300a47e0 msgr2=0x7f7630131250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:44:54.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f7636d60640 1 --2- 172.21.15.62:0/2297685527 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76300a47e0 0x7f7630131250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:54.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f7636d60640 1 -- 172.21.15.62:0/2297685527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76300a5100 msgr2=0x7f7630139040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:54.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f7636d60640 1 --2- 172.21.15.62:0/2297685527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76300a5100 0x7f7630139040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:54.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f7636d60640 10 monclient: _finish_hunting 0 2024-04-19T01:44:54.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f7636d60640 1 monclient: found mon.noname-c 2024-04-19T01:44:54.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f7636d60640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:44:54.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f7636d60640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:44:54.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f7636d60640 1 -- 172.21.15.62:0/2297685527 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f763013ace0 con 0x7f76301355f0 2024-04-19T01:44:54.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.841+0000 7f7635d5e640 1 --2- 172.21.15.62:0/2297685527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76300a5100 0x7f7630139040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:44:54.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.845+0000 7f7636d60640 1 -- 172.21.15.62:0/2297685527 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f76301355f0 msgr2=0x7f7630139580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:54.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.845+0000 7f7636d60640 1 --2- 172.21.15.62:0/2297685527 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f76301355f0 0x7f7630139580 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f762800bdc0 tx=0x7f7628009eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:54.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.845+0000 7f7626ffd640 1 -- 172.21.15.62:0/2297685527 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76280066f0 con 0x7f76301355f0 2024-04-19T01:44:54.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.845+0000 7f7626ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:54.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.845+0000 7f7626ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:44:54.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.845+0000 7f7626ffd640 10 monclient: dump: 2024-04-19T01:44:54.850 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:54.850 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:54.850 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:54.850 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:54.851 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:54.851 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:54.851 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:54.851 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:54.851 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:54.851 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:54.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.845+0000 7f7626ffd640 10 monclient: _finish_auth 0 2024-04-19T01:44:54.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.845+0000 7f7626ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:44:54.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.845+0000 7f7626ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:44:54.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.845+0000 7f7626ffd640 1 -- 172.21.15.62:0/2297685527 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76280068b0 con 0x7f76301355f0 2024-04-19T01:44:54.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.845+0000 7f7626ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:44:54.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.845+0000 7f7626ffd640 1 -- 172.21.15.62:0/2297685527 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7628015440 con 0x7f76301355f0 2024-04-19T01:44:54.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.845+0000 7f76387c3640 5 monclient: authenticate success, global_id 4826 2024-04-19T01:44:54.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.845+0000 7f76387c3640 10 monclient: _renew_subs 2024-04-19T01:44:54.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.845+0000 7f76387c3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:44:54.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.845+0000 7f76387c3640 1 -- 172.21.15.62:0/2297685527 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76301cb920 con 0x7f76301355f0 2024-04-19T01:44:54.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.845+0000 7f7626ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:44:54.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.845+0000 7f7626ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:44:54.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.845+0000 7f7626ffd640 10 monclient: dump: 2024-04-19T01:44:54.852 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:44:54.852 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:54.852 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:54.852 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:44:54.852 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:44:54.852 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:54.852 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:44:54.852 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:44:54.852 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:44:54.853 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:54.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.845+0000 7f7634d5c640 4 set_mon_vals no callback set 2024-04-19T01:44:54.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.845+0000 7f76387c3640 10 monclient: _renew_subs 2024-04-19T01:44:54.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.845+0000 7f76387c3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:44:54.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.845+0000 7f76387c3640 1 -- 172.21.15.62:0/2297685527 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f763013c730 con 0x7f76301355f0 2024-04-19T01:44:54.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.845+0000 7f7626ffd640 1 -- 172.21.15.62:0/2297685527 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f762801e3f0 con 0x7f76301355f0 2024-04-19T01:44:54.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.845+0000 7f76387c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:54.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.845+0000 7f76387c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:54.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.845+0000 7f76387c3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:44:54.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.845+0000 7f7626ffd640 1 --2- 172.21.15.62:0/2297685527 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f7614036450 0x7f7614038910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:54.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.845+0000 7f76387c3640 1 -- 172.21.15.62:0/2297685527 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76300692b0 con 0x7f76301355f0 2024-04-19T01:44:54.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.845+0000 7f7626ffd640 1 -- 172.21.15.62:0/2297685527 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(118..118 src has 1..118) ==== 4738+0+0 (secure 0 0 0) 0x7f762804a2c0 con 0x7f76301355f0 2024-04-19T01:44:54.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.845+0000 7f763655f640 1 --2- 172.21.15.62:0/2297685527 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f7614036450 0x7f7614038910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:54.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.845+0000 7f763655f640 10 monclient: get_auth_request con 0x7f7614036450 auth_method 0 2024-04-19T01:44:54.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.849+0000 7f763655f640 1 --2- 172.21.15.62:0/2297685527 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f7614036450 0x7f7614038910 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f762c0ab000 tx=0x7f762c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:44:54.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.849+0000 7f7626ffd640 1 -- 172.21.15.62:0/2297685527 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7628048ce0 con 0x7f76301355f0 2024-04-19T01:44:54.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.849+0000 7f7626ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:44:54.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:54.849+0000 7f7626ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:44:55.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:55.077+0000 7f76387c3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:44:55.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:55.077+0000 7f76387c3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:44:55.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:55.077+0000 7f76387c3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:44:55.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:55.077+0000 7f76387c3640 1 -- 172.21.15.62:0/2297685527 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f76300a45c0 con 0x7f76301355f0 2024-04-19T01:44:55.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:55.077+0000 7f7626ffd640 1 -- 172.21.15.62:0/2297685527 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f7628024080 con 0x7f76301355f0 2024-04-19T01:44:55.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:55.077+0000 7f7626ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:44:55.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:55.077+0000 7f7626ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:44:55.075 INFO:teuthology.orchestra.run.smithi062.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","modified":"2024-04-19T01:41:38.497795Z","created":"2024-04-19T01:41:38.497795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:44:55.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:55.077+0000 7f76387c3640 1 -- 172.21.15.62:0/2297685527 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f7614036450 msgr2=0x7f7614038910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:55.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:55.077+0000 7f76387c3640 1 --2- 172.21.15.62:0/2297685527 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f7614036450 0x7f7614038910 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f762c0ab000 tx=0x7f762c008040 comp rx=0 tx=0).stop 2024-04-19T01:44:55.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:55.077+0000 7f76387c3640 10 monclient: shutdown 2024-04-19T01:44:55.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:55.077+0000 7f76387c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:44:55.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:55.077+0000 7f76387c3640 1 -- 172.21.15.62:0/2297685527 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f76301355f0 msgr2=0x7f7630139580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:44:55.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:55.077+0000 7f76387c3640 1 --2- 172.21.15.62:0/2297685527 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f76301355f0 0x7f7630139580 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f762800bdc0 tx=0x7f7628009eb0 comp rx=0 tx=0).stop 2024-04-19T01:44:55.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:55.081+0000 7f76387c3640 1 -- 172.21.15.62:0/2297685527 shutdown_connections 2024-04-19T01:44:55.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:55.081+0000 7f76387c3640 1 --2- 172.21.15.62:0/2297685527 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f7614036450 0x7f7614038910 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:55.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:55.081+0000 7f76387c3640 1 --2- 172.21.15.62:0/2297685527 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f76301355f0 0x7f7630139580 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:55.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:55.081+0000 7f76387c3640 1 --2- 172.21.15.62:0/2297685527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76300a5100 0x7f7630139040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:55.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:55.081+0000 7f76387c3640 1 --2- 172.21.15.62:0/2297685527 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76300a47e0 0x7f7630131250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:44:55.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:55.081+0000 7f76387c3640 1 -- 172.21.15.62:0/2297685527 >> 172.21.15.62:0/2297685527 conn(0x7f76300a23d0 msgr2=0x7f76300a11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:44:55.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:55.081+0000 7f76387c3640 1 -- 172.21.15.62:0/2297685527 shutdown_connections 2024-04-19T01:44:55.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:55.081+0000 7f76387c3640 1 -- 172.21.15.62:0/2297685527 wait complete. 2024-04-19T01:44:58.094 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:44:58.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.229+0000 7fe1a43bf640 10 monclient: get_monmap_and_config 2024-04-19T01:44:58.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.229+0000 7fe1a43bf640 10 monclient: build_initial_monmap 2024-04-19T01:44:58.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.229+0000 7fe1a43bf640 10 monclient: monmap: 2024-04-19T01:44:58.226 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:44:58.226 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:44:58.226 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:44:58.231553+0000 2024-04-19T01:44:58.227 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:44:58.231553+0000 2024-04-19T01:44:58.227 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:44:58.227 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:44:58.227 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:44:58.227 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:44:58.227 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:44:58.227 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:44:58.227 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:44:58.227 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:44:58.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.229+0000 7fe1a43bf640 1 Processor -- start 2024-04-19T01:44:58.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.229+0000 7fe1a43bf640 1 -- start start 2024-04-19T01:44:58.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.229+0000 7fe1a43bf640 10 monclient: init 2024-04-19T01:44:58.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.229+0000 7fe1a43bf640 10 monclient: _reopen_session rank -1 2024-04-19T01:44:58.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.229+0000 7fe1a43bf640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-04-19T01:44:58.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.229+0000 7fe1a43bf640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe19c137780 0x7fe19c137b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:44:58.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.229+0000 7fe1a43bf640 10 monclient(hunting): picked mon.noname-b con 0x7fe19c137780 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:44:58.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.229+0000 7fe1a43bf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe19c1380a0 addr v1:172.21.15.71:6789/0 2024-04-19T01:44:58.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.229+0000 7fe1a43bf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe19c0a60b0 addr v1:172.21.15.62:6789/0 2024-04-19T01:44:58.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.229+0000 7fe1a43bf640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe19c09c2e0 con 0x7fe19c0a60b0 2024-04-19T01:44:58.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.229+0000 7fe1a43bf640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe19c0a53f0 con 0x7fe19c1380a0 2024-04-19T01:44:58.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.229+0000 7fe1a43bf640 10 monclient(hunting): start opening mon connection 2024-04-19T01:44:58.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.229+0000 7fe1a43bf640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fe19c09bd20 con 0x7fe19c137780 2024-04-19T01:44:58.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.229+0000 7fe1a43bf640 10 monclient(hunting): _renew_subs 2024-04-19T01:44:58.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.229+0000 7fe1a43bf640 10 monclient(hunting): authenticate will time out at 1084.051562s 2024-04-19T01:44:58.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.229+0000 7fe1a215b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe19c137780 0x7fe19c137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:58.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.229+0000 7fe1a195a640 1 -- >> v1:172.21.15.71:6789/0 conn(0x7fe19c1380a0 legacy=0x7fe19c0a45d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-04-19T01:44:58.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.229+0000 7fe1a295c640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fe19c0a60b0 0x7fe19c0a4ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:50042/0 (socket says 172.21.15.62:50042) 2024-04-19T01:44:58.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.229+0000 7fe1a295c640 1 -- 172.21.15.62:0/2120878631 learned_addr learned my addr 172.21.15.62:0/2120878631 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:44:58.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.229+0000 7fe1a215b640 10 monclient(hunting): get_auth_request con 0x7fe19c137780 auth_method 0 2024-04-19T01:44:58.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.229+0000 7fe1a215b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:58.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.229+0000 7fe1a215b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:58.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.229+0000 7fe1a215b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:44:58.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.229+0000 7fe1a215b640 1 -- 172.21.15.62:0/2120878631 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe19c137780 msgr2=0x7fe19c137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:44:58.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.229+0000 7fe1a215b640 1 -- 172.21.15.62:0/2120878631 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe19c137780 msgr2=0x7fe19c137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:58.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.229+0000 7fe1a215b640 1 --2- 172.21.15.62:0/2120878631 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe19c137780 0x7fe19c137b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:58.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.229+0000 7fe1a215b640 1 --2- 172.21.15.62:0/2120878631 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe19c137780 0x7fe19c137b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:44:58.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.233+0000 7fe1a295c640 1 -- 172.21.15.62:0/2120878631 >> v1:172.21.15.62:6789/0 conn(0x7fe19c0a60b0 legacy=0x7fe19c0a4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:44:58.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.433+0000 7fe1a195a640 1 -- 172.21.15.62:0/2120878631 >> v1:172.21.15.71:6789/0 conn(0x7fe19c1380a0 legacy=0x7fe19c0a45d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-04-19T01:44:58.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.433+0000 7fe1a215b640 1 --2- 172.21.15.62:0/2120878631 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe19c137780 0x7fe19c137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:58.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.433+0000 7fe1a215b640 10 monclient(hunting): get_auth_request con 0x7fe19c137780 auth_method 0 2024-04-19T01:44:58.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.433+0000 7fe1a215b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:58.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.433+0000 7fe1a215b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:58.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.433+0000 7fe1a215b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:58.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.433+0000 7fe1a215b640 1 -- 172.21.15.62:0/2120878631 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe19c137780 msgr2=0x7fe19c137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:44:58.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.433+0000 7fe1a215b640 1 -- 172.21.15.62:0/2120878631 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe19c137780 msgr2=0x7fe19c137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:58.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.433+0000 7fe1a215b640 1 --2- 172.21.15.62:0/2120878631 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe19c137780 0x7fe19c137b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:58.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.433+0000 7fe1a215b640 1 --2- 172.21.15.62:0/2120878631 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe19c137780 0x7fe19c137b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:44:58.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.837+0000 7fe1a195a640 1 -- 172.21.15.62:0/2120878631 >> v1:172.21.15.71:6789/0 conn(0x7fe19c1380a0 legacy=0x7fe19c0a45d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-04-19T01:44:58.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.837+0000 7fe1a215b640 1 --2- 172.21.15.62:0/2120878631 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe19c137780 0x7fe19c137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:58.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.837+0000 7fe1a215b640 10 monclient(hunting): get_auth_request con 0x7fe19c137780 auth_method 0 2024-04-19T01:44:58.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.837+0000 7fe1a215b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:58.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.837+0000 7fe1a215b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:58.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.837+0000 7fe1a215b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:58.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.837+0000 7fe1a215b640 1 -- 172.21.15.62:0/2120878631 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe19c137780 msgr2=0x7fe19c137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:44:58.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.837+0000 7fe1a215b640 1 -- 172.21.15.62:0/2120878631 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe19c137780 msgr2=0x7fe19c137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:58.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.837+0000 7fe1a215b640 1 --2- 172.21.15.62:0/2120878631 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe19c137780 0x7fe19c137b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:58.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:58.837+0000 7fe1a215b640 1 --2- 172.21.15.62:0/2120878631 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe19c137780 0x7fe19c137b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:44:59.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:59.641+0000 7fe1a195a640 1 -- 172.21.15.62:0/2120878631 >> v1:172.21.15.71:6789/0 conn(0x7fe19c1380a0 legacy=0x7fe19c0a45d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-04-19T01:44:59.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:59.641+0000 7fe1a215b640 1 --2- 172.21.15.62:0/2120878631 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe19c137780 0x7fe19c137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:44:59.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:59.641+0000 7fe1a215b640 10 monclient(hunting): get_auth_request con 0x7fe19c137780 auth_method 0 2024-04-19T01:44:59.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:59.641+0000 7fe1a215b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:44:59.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:59.641+0000 7fe1a215b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:44:59.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:59.641+0000 7fe1a215b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:44:59.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:59.641+0000 7fe1a215b640 1 -- 172.21.15.62:0/2120878631 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe19c137780 msgr2=0x7fe19c137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:44:59.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:59.641+0000 7fe1a215b640 1 -- 172.21.15.62:0/2120878631 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe19c137780 msgr2=0x7fe19c137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:44:59.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:59.641+0000 7fe1a215b640 1 --2- 172.21.15.62:0/2120878631 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe19c137780 0x7fe19c137b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:44:59.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:44:59.641+0000 7fe1a215b640 1 --2- 172.21.15.62:0/2120878631 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe19c137780 0x7fe19c137b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:45:01.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.233+0000 7fe193fff640 10 monclient(hunting): tick 2024-04-19T01:45:01.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.233+0000 7fe193fff640 10 monclient(hunting): _check_auth_tickets 2024-04-19T01:45:01.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.233+0000 7fe193fff640 1 monclient(hunting): continuing hunt 2024-04-19T01:45:01.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.233+0000 7fe193fff640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T01:45:01.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.233+0000 7fe193fff640 1 -- 172.21.15.62:0/2120878631 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe19c137780 msgr2=0x7fe19c137b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:45:01.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.233+0000 7fe193fff640 1 --2- 172.21.15.62:0/2120878631 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe19c137780 0x7fe19c137b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:45:01.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.233+0000 7fe193fff640 1 -- 172.21.15.62:0/2120878631 >> v1:172.21.15.71:6789/0 conn(0x7fe19c1380a0 legacy=0x7fe19c0a45d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:45:01.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.233+0000 7fe193fff640 1 -- 172.21.15.62:0/2120878631 >> v1:172.21.15.62:6789/0 conn(0x7fe19c0a60b0 legacy=0x7fe19c0a4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:45:01.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.233+0000 7fe193fff640 10 monclient: _add_conns ranks=[3,4] 2024-04-19T01:45:01.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.233+0000 7fe193fff640 1 --2- 172.21.15.62:0/2120878631 >> v2:172.21.15.62:3300/0 conn(0x7fe180003760 0x7fe180005be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:45:01.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.233+0000 7fe193fff640 10 monclient(hunting): picked mon.noname-a con 0x7fe180003760 addr v2:172.21.15.62:3300/0 2024-04-19T01:45:01.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.233+0000 7fe193fff640 1 --2- 172.21.15.62:0/2120878631 >> v2:172.21.15.71:3300/0 conn(0x7fe1800062d0 0x7fe180008790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:45:01.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.233+0000 7fe193fff640 10 monclient(hunting): picked mon.noname-c con 0x7fe1800062d0 addr v2:172.21.15.71:3300/0 2024-04-19T01:45:01.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.233+0000 7fe193fff640 10 monclient(hunting): start opening mon connection 2024-04-19T01:45:01.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.233+0000 7fe193fff640 1 -- 172.21.15.62:0/2120878631 --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fe19c09bd20 con 0x7fe180003760 2024-04-19T01:45:01.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.233+0000 7fe193fff640 10 monclient(hunting): start opening mon connection 2024-04-19T01:45:01.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.233+0000 7fe193fff640 1 -- 172.21.15.62:0/2120878631 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fe180008e50 con 0x7fe1800062d0 2024-04-19T01:45:01.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.233+0000 7fe193fff640 10 monclient(hunting): _renew_subs 2024-04-19T01:45:01.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.233+0000 7fe1a195a640 1 -- 172.21.15.62:0/2120878631 >> v2:172.21.15.71:3300/0 conn(0x7fe1800062d0 msgr2=0x7fe180008790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-04-19T01:45:01.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.233+0000 7fe1a195a640 1 --2- 172.21.15.62:0/2120878631 >> v2:172.21.15.71:3300/0 conn(0x7fe1800062d0 0x7fe180008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:45:01.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.233+0000 7fe1a215b640 1 --2- 172.21.15.62:0/2120878631 >> v2:172.21.15.62:3300/0 conn(0x7fe180003760 0x7fe180005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:45:01.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.233+0000 7fe1a215b640 10 monclient(hunting): get_auth_request con 0x7fe180003760 auth_method 0 2024-04-19T01:45:01.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.233+0000 7fe1a215b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:45:01.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.233+0000 7fe1a215b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:45:01.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.233+0000 7fe1a215b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:45:01.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.233+0000 7fe1a215b640 1 -- 172.21.15.62:0/2120878631 >> v2:172.21.15.62:3300/0 conn(0x7fe180003760 msgr2=0x7fe180005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:45:01.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.233+0000 7fe1a215b640 1 -- 172.21.15.62:0/2120878631 >> v2:172.21.15.62:3300/0 conn(0x7fe180003760 msgr2=0x7fe180005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:45:01.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.233+0000 7fe1a215b640 1 --2- 172.21.15.62:0/2120878631 >> v2:172.21.15.62:3300/0 conn(0x7fe180003760 0x7fe180005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:45:01.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.233+0000 7fe1a215b640 1 --2- 172.21.15.62:0/2120878631 >> v2:172.21.15.62:3300/0 conn(0x7fe180003760 0x7fe180005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:45:01.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.436+0000 7fe1a195a640 1 -- 172.21.15.62:0/2120878631 >> v2:172.21.15.71:3300/0 conn(0x7fe1800062d0 msgr2=0x7fe180008790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-04-19T01:45:01.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.436+0000 7fe1a195a640 1 --2- 172.21.15.62:0/2120878631 >> v2:172.21.15.71:3300/0 conn(0x7fe1800062d0 0x7fe180008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:45:01.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.436+0000 7fe1a215b640 1 --2- 172.21.15.62:0/2120878631 >> v2:172.21.15.62:3300/0 conn(0x7fe180003760 0x7fe180005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:45:01.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.436+0000 7fe1a215b640 10 monclient(hunting): get_auth_request con 0x7fe180003760 auth_method 0 2024-04-19T01:45:01.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.436+0000 7fe1a215b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:45:01.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.436+0000 7fe1a215b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:45:01.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.436+0000 7fe1a215b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:45:01.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.436+0000 7fe1a215b640 1 -- 172.21.15.62:0/2120878631 >> v2:172.21.15.62:3300/0 conn(0x7fe180003760 msgr2=0x7fe180005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:45:01.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.436+0000 7fe1a215b640 1 -- 172.21.15.62:0/2120878631 >> v2:172.21.15.62:3300/0 conn(0x7fe180003760 msgr2=0x7fe180005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:45:01.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.436+0000 7fe1a215b640 1 --2- 172.21.15.62:0/2120878631 >> v2:172.21.15.62:3300/0 conn(0x7fe180003760 0x7fe180005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:45:01.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.436+0000 7fe1a215b640 1 --2- 172.21.15.62:0/2120878631 >> v2:172.21.15.62:3300/0 conn(0x7fe180003760 0x7fe180005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:45:01.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.840+0000 7fe1a195a640 1 -- 172.21.15.62:0/2120878631 >> v2:172.21.15.71:3300/0 conn(0x7fe1800062d0 msgr2=0x7fe180008790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-04-19T01:45:01.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.840+0000 7fe1a195a640 1 --2- 172.21.15.62:0/2120878631 >> v2:172.21.15.71:3300/0 conn(0x7fe1800062d0 0x7fe180008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:45:01.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.840+0000 7fe1a215b640 1 --2- 172.21.15.62:0/2120878631 >> v2:172.21.15.62:3300/0 conn(0x7fe180003760 0x7fe180005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:45:01.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.840+0000 7fe1a215b640 10 monclient(hunting): get_auth_request con 0x7fe180003760 auth_method 0 2024-04-19T01:45:01.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.840+0000 7fe1a215b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:45:01.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.840+0000 7fe1a215b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:45:01.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.840+0000 7fe1a215b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:45:01.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.840+0000 7fe1a215b640 1 -- 172.21.15.62:0/2120878631 >> v2:172.21.15.62:3300/0 conn(0x7fe180003760 msgr2=0x7fe180005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:45:01.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.840+0000 7fe1a215b640 1 -- 172.21.15.62:0/2120878631 >> v2:172.21.15.62:3300/0 conn(0x7fe180003760 msgr2=0x7fe180005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:45:01.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.840+0000 7fe1a215b640 1 --2- 172.21.15.62:0/2120878631 >> v2:172.21.15.62:3300/0 conn(0x7fe180003760 0x7fe180005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:45:01.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:01.840+0000 7fe1a215b640 1 --2- 172.21.15.62:0/2120878631 >> v2:172.21.15.62:3300/0 conn(0x7fe180003760 0x7fe180005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:45:02.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.644+0000 7fe1a195a640 1 -- 172.21.15.62:0/2120878631 >> v2:172.21.15.71:3300/0 conn(0x7fe1800062d0 msgr2=0x7fe180008790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-04-19T01:45:02.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.644+0000 7fe1a195a640 1 --2- 172.21.15.62:0/2120878631 >> v2:172.21.15.71:3300/0 conn(0x7fe1800062d0 0x7fe180008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:45:02.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.644+0000 7fe1a215b640 1 --2- 172.21.15.62:0/2120878631 >> v2:172.21.15.62:3300/0 conn(0x7fe180003760 0x7fe180005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:45:02.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.644+0000 7fe1a215b640 10 monclient(hunting): get_auth_request con 0x7fe180003760 auth_method 0 2024-04-19T01:45:02.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.644+0000 7fe1a215b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:45:02.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.644+0000 7fe1a215b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:45:02.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.644+0000 7fe1a215b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:45:02.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.644+0000 7fe1a215b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:45:02.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.644+0000 7fe1a215b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:45:02.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.644+0000 7fe1a215b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:45:02.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.644+0000 7fe1a215b640 10 monclient(hunting): handle_auth_done global_id 14232 payload 274 2024-04-19T01:45:02.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.644+0000 7fe1a215b640 1 -- 172.21.15.62:0/2120878631 >> v2:172.21.15.71:3300/0 conn(0x7fe1800062d0 msgr2=0x7fe180008790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:45:02.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.644+0000 7fe1a215b640 1 --2- 172.21.15.62:0/2120878631 >> v2:172.21.15.71:3300/0 conn(0x7fe1800062d0 0x7fe180008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:45:02.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.644+0000 7fe1a215b640 10 monclient: _finish_hunting 0 2024-04-19T01:45:02.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.644+0000 7fe1a215b640 1 monclient: found mon.noname-a 2024-04-19T01:45:02.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.644+0000 7fe1a215b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:45:02.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.644+0000 7fe1a215b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:45:02.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.644+0000 7fe1a215b640 1 -- 172.21.15.62:0/2120878631 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe19c09c000 con 0x7fe180003760 2024-04-19T01:45:02.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.652+0000 7fe1a215b640 1 -- 172.21.15.62:0/2120878631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe180003760 msgr2=0x7fe180005be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:45:02.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.652+0000 7fe1a215b640 1 --2- 172.21.15.62:0/2120878631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe180003760 0x7fe180005be0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe1800062a0 tx=0x7fe19801e3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e14691a43af6214 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:45:02.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.652+0000 7fe1a1159640 1 -- 172.21.15.62:0/2120878631 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1980b4070 con 0x7fe180003760 2024-04-19T01:45:02.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.652+0000 7fe1a1159640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:45:02.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.652+0000 7fe1a1159640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:45:02.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.652+0000 7fe1a1159640 10 monclient: dump: 2024-04-19T01:45:02.650 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:45:02.650 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:45:02.650 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:45:02.650 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:45:02.651 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:45:02.651 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:45:02.651 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:45:02.651 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:45:02.651 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:45:02.651 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:45:02.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.652+0000 7fe1a1159640 10 monclient: _finish_auth 0 2024-04-19T01:45:02.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.652+0000 7fe1a1159640 10 monclient: _check_auth_tickets 2024-04-19T01:45:02.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.652+0000 7fe1a1159640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:45:02.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.652+0000 7fe1a1159640 1 -- 172.21.15.62:0/2120878631 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1980af410 con 0x7fe180003760 2024-04-19T01:45:02.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.652+0000 7fe1a1159640 10 monclient: handle_config config(0 keys) 2024-04-19T01:45:02.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.652+0000 7fe1a1159640 1 -- 172.21.15.62:0/2120878631 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1980af5e0 con 0x7fe180003760 2024-04-19T01:45:02.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.652+0000 7fe1a1159640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:45:02.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.652+0000 7fe1a1159640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:45:02.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.652+0000 7fe1a1159640 10 monclient: dump: 2024-04-19T01:45:02.652 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:45:02.652 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:45:02.652 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:45:02.652 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:45:02.652 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:45:02.652 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:45:02.652 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:45:02.652 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:45:02.652 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:45:02.652 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:45:02.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.652+0000 7fe1a43bf640 5 monclient: authenticate success, global_id 14232 2024-04-19T01:45:02.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.652+0000 7fe1a43bf640 10 monclient: get_monmap_and_config success 2024-04-19T01:45:02.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.652+0000 7fe1a43bf640 4 set_mon_vals no callback set 2024-04-19T01:45:02.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.652+0000 7fe1a43bf640 10 monclient: shutdown 2024-04-19T01:45:02.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.652+0000 7fe1a43bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:45:02.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.652+0000 7fe1a43bf640 1 -- 172.21.15.62:0/2120878631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe180003760 msgr2=0x7fe180005be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:45:02.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.652+0000 7fe1a43bf640 1 --2- 172.21.15.62:0/2120878631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe180003760 0x7fe180005be0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe1800062a0 tx=0x7fe19801e3a0 comp rx=0 tx=0).stop 2024-04-19T01:45:02.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.652+0000 7fe1a295c640 1 -- 172.21.15.62:0/2120878631 reap_dead start 2024-04-19T01:45:02.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.652+0000 7fe1a43bf640 1 -- 172.21.15.62:0/2120878631 shutdown_connections 2024-04-19T01:45:02.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.652+0000 7fe1a43bf640 1 --2- 172.21.15.62:0/2120878631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe180003760 0x7fe180005be0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:45:02.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.652+0000 7fe1a43bf640 1 -- 172.21.15.62:0/2120878631 >> 172.21.15.62:0/2120878631 conn(0x7fe19c0aa1d0 msgr2=0x7fe19c0a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:45:02.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.652+0000 7fe1a43bf640 1 -- 172.21.15.62:0/2120878631 shutdown_connections 2024-04-19T01:45:02.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.652+0000 7fe1a43bf640 1 -- 172.21.15.62:0/2120878631 wait complete. 2024-04-19T01:45:02.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a43bf640 10 monclient: build_initial_monmap 2024-04-19T01:45:02.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a43bf640 10 monclient: monmap: 2024-04-19T01:45:02.657 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:45:02.657 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:45:02.657 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:45:02.657 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:45:02.658 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:45:02.658 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:45:02.658 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:45:02.658 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:45:02.658 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:45:02.658 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:45:02.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a43bf640 1 Processor -- start 2024-04-19T01:45:02.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a43bf640 1 -- start start 2024-04-19T01:45:02.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a43bf640 10 monclient: init 2024-04-19T01:45:02.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a43bf640 10 monclient: _reopen_session rank -1 2024-04-19T01:45:02.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a43bf640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-19T01:45:02.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a43bf640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe180003760 0x7fe19c1cf980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:45:02.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a43bf640 10 monclient(hunting): picked mon.noname-b con 0x7fe180003760 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:45:02.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a43bf640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe19c13f8a0 0x7fe19c13fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:45:02.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a43bf640 10 monclient(hunting): picked mon.noname-a con 0x7fe19c13f8a0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:45:02.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a43bf640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe19c09f9c0 0x7fe19c09fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:45:02.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a43bf640 10 monclient(hunting): picked mon.noname-c con 0x7fe19c09f9c0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:45:02.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a43bf640 10 monclient(hunting): start opening mon connection 2024-04-19T01:45:02.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a43bf640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe19c143300 con 0x7fe19c13f8a0 2024-04-19T01:45:02.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a43bf640 10 monclient(hunting): start opening mon connection 2024-04-19T01:45:02.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a43bf640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe19c143180 con 0x7fe180003760 2024-04-19T01:45:02.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a43bf640 10 monclient(hunting): start opening mon connection 2024-04-19T01:45:02.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a43bf640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fe19c143480 con 0x7fe19c09f9c0 2024-04-19T01:45:02.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a43bf640 10 monclient(hunting): _renew_subs 2024-04-19T01:45:02.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a43bf640 10 monclient(hunting): authenticate will time out at 1088.477586s 2024-04-19T01:45:02.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a215b640 1 -- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe180003760 msgr2=0x7fe19c1cf980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-04-19T01:45:02.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a215b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe180003760 0x7fe19c1cf980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:45:02.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a295c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe19c09f9c0 0x7fe19c09fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:45:02.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a195a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe19c13f8a0 0x7fe19c13fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:45:02.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a295c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe19c09f9c0 0x7fe19c09fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:58060/0 (socket says 172.21.15.62:58060) 2024-04-19T01:45:02.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a295c640 1 -- 172.21.15.62:0/3821842415 learned_addr learned my addr 172.21.15.62:0/3821842415 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:45:02.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a295c640 10 monclient(hunting): get_auth_request con 0x7fe19c09f9c0 auth_method 0 2024-04-19T01:45:02.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a295c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:45:02.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a295c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:45:02.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a295c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:45:02.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a195a640 10 monclient(hunting): get_auth_request con 0x7fe19c13f8a0 auth_method 0 2024-04-19T01:45:02.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a195a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:45:02.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a195a640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:45:02.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a195a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:45:02.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a295c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:45:02.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a295c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:45:02.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a295c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:45:02.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a195a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:45:02.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a195a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:45:02.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a195a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:45:02.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a295c640 10 monclient(hunting): handle_auth_done global_id 4829 payload 824 2024-04-19T01:45:02.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a295c640 1 -- 172.21.15.62:0/3821842415 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe180003760 msgr2=0x7fe19c1cf980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:45:02.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a295c640 1 --2- 172.21.15.62:0/3821842415 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe180003760 0x7fe19c1cf980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:45:02.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a295c640 1 -- 172.21.15.62:0/3821842415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe19c13f8a0 msgr2=0x7fe19c13fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:45:02.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a295c640 1 --2- 172.21.15.62:0/3821842415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe19c13f8a0 0x7fe19c13fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:45:02.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a295c640 10 monclient: _finish_hunting 0 2024-04-19T01:45:02.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a295c640 1 monclient: found mon.noname-c 2024-04-19T01:45:02.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a295c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:45:02.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a295c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:45:02.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.656+0000 7fe1a295c640 1 -- 172.21.15.62:0/3821842415 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe19c0a3420 con 0x7fe19c09f9c0 2024-04-19T01:45:02.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.660+0000 7fe1a295c640 1 -- 172.21.15.62:0/3821842415 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe19c09f9c0 msgr2=0x7fe19c09fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:45:02.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.660+0000 7fe1a295c640 1 --2- 172.21.15.62:0/3821842415 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe19c09f9c0 0x7fe19c09fe20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe180008df0 tx=0x7fe19400c760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:45:02.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.660+0000 7fe192ffd640 1 -- 172.21.15.62:0/3821842415 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe194007110 con 0x7fe19c09f9c0 2024-04-19T01:45:02.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.660+0000 7fe192ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:45:02.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.660+0000 7fe192ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:45:02.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.660+0000 7fe192ffd640 10 monclient: dump: 2024-04-19T01:45:02.665 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:45:02.665 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:45:02.666 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:45:02.666 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:45:02.666 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:45:02.666 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:45:02.666 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:45:02.666 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:45:02.666 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:45:02.666 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:45:02.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.660+0000 7fe192ffd640 10 monclient: _finish_auth 0 2024-04-19T01:45:02.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.660+0000 7fe192ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:45:02.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.660+0000 7fe192ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:45:02.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.660+0000 7fe192ffd640 1 -- 172.21.15.62:0/3821842415 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1940072b0 con 0x7fe19c09f9c0 2024-04-19T01:45:02.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.660+0000 7fe192ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:45:02.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.660+0000 7fe192ffd640 1 -- 172.21.15.62:0/3821842415 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe194007660 con 0x7fe19c09f9c0 2024-04-19T01:45:02.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.660+0000 7fe192ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:45:02.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.660+0000 7fe192ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:45:02.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.660+0000 7fe192ffd640 10 monclient: dump: 2024-04-19T01:45:02.667 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:45:02.667 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:45:02.667 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:45:02.667 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:45:02.667 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:45:02.667 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:45:02.667 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:45:02.667 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:45:02.667 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:45:02.667 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:45:02.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.660+0000 7fe1a43bf640 5 monclient: authenticate success, global_id 4829 2024-04-19T01:45:02.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.660+0000 7fe1a43bf640 10 monclient: _renew_subs 2024-04-19T01:45:02.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.660+0000 7fe1a43bf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:45:02.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.660+0000 7fe1a0958640 4 set_mon_vals no callback set 2024-04-19T01:45:02.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.660+0000 7fe1a43bf640 1 -- 172.21.15.62:0/3821842415 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe19c131630 con 0x7fe19c09f9c0 2024-04-19T01:45:02.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.660+0000 7fe1a43bf640 10 monclient: _renew_subs 2024-04-19T01:45:02.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.660+0000 7fe1a43bf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:45:02.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.660+0000 7fe1a43bf640 1 -- 172.21.15.62:0/3821842415 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe19c139320 con 0x7fe19c09f9c0 2024-04-19T01:45:02.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.660+0000 7fe1a43bf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:45:02.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.660+0000 7fe1a43bf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:45:02.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.660+0000 7fe1a43bf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:45:02.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.660+0000 7fe1a43bf640 1 -- 172.21.15.62:0/3821842415 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe19c068ab0 con 0x7fe19c09f9c0 2024-04-19T01:45:02.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.660+0000 7fe192ffd640 1 -- 172.21.15.62:0/3821842415 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fe194011070 con 0x7fe19c09f9c0 2024-04-19T01:45:02.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.660+0000 7fe192ffd640 1 --2- 172.21.15.62:0/3821842415 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fe180037310 0x7fe1800397d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:45:02.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.660+0000 7fe192ffd640 1 -- 172.21.15.62:0/3821842415 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(118..118 src has 1..118) ==== 4738+0+0 (secure 0 0 0) 0x7fe194048240 con 0x7fe19c09f9c0 2024-04-19T01:45:02.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.660+0000 7fe1a215b640 1 --2- 172.21.15.62:0/3821842415 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fe180037310 0x7fe1800397d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:45:02.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.660+0000 7fe1a215b640 10 monclient: get_auth_request con 0x7fe180037310 auth_method 0 2024-04-19T01:45:02.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.664+0000 7fe1a215b640 1 --2- 172.21.15.62:0/3821842415 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fe180037310 0x7fe1800397d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe1980ae000 tx=0x7fe1980a8000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:45:02.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.664+0000 7fe192ffd640 1 -- 172.21.15.62:0/3821842415 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe194019060 con 0x7fe19c09f9c0 2024-04-19T01:45:02.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.664+0000 7fe192ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:45:02.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.664+0000 7fe192ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:45:02.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.916+0000 7fe1a43bf640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:45:02.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.916+0000 7fe1a43bf640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:45:02.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.916+0000 7fe1a43bf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:45:02.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.916+0000 7fe1a43bf640 1 -- 172.21.15.62:0/3821842415 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe19c0a9410 con 0x7fe19c09f9c0 2024-04-19T01:45:02.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.916+0000 7fe192ffd640 1 -- 172.21.15.62:0/3821842415 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7fe19c0a3420 con 0x7fe19c09f9c0 2024-04-19T01:45:02.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.916+0000 7fe192ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:45:02.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.916+0000 7fe192ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:45:02.915 INFO:teuthology.orchestra.run.smithi062.stdout:{"election_epoch":32,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","modified":"2024-04-19T01:41:38.497795Z","created":"2024-04-19T01:41:38.497795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:45:02.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.920+0000 7fe1a43bf640 1 -- 172.21.15.62:0/3821842415 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fe180037310 msgr2=0x7fe1800397d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:45:02.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.920+0000 7fe1a43bf640 1 --2- 172.21.15.62:0/3821842415 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fe180037310 0x7fe1800397d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe1980ae000 tx=0x7fe1980a8000 comp rx=0 tx=0).stop 2024-04-19T01:45:02.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.920+0000 7fe1a43bf640 10 monclient: shutdown 2024-04-19T01:45:02.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.920+0000 7fe1a43bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:45:02.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.920+0000 7fe1a43bf640 1 -- 172.21.15.62:0/3821842415 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe19c09f9c0 msgr2=0x7fe19c09fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:45:02.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.920+0000 7fe1a43bf640 1 --2- 172.21.15.62:0/3821842415 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe19c09f9c0 0x7fe19c09fe20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe180008df0 tx=0x7fe19400c760 comp rx=0 tx=0).stop 2024-04-19T01:45:02.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.920+0000 7fe1a43bf640 1 -- 172.21.15.62:0/3821842415 shutdown_connections 2024-04-19T01:45:02.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.920+0000 7fe1a43bf640 1 --2- 172.21.15.62:0/3821842415 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fe180037310 0x7fe1800397d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:45:02.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.920+0000 7fe1a43bf640 1 --2- 172.21.15.62:0/3821842415 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe19c09f9c0 0x7fe19c09fe20 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:45:02.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.920+0000 7fe1a43bf640 1 --2- 172.21.15.62:0/3821842415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe19c13f8a0 0x7fe19c13fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:45:02.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.920+0000 7fe1a43bf640 1 --2- 172.21.15.62:0/3821842415 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe180003760 0x7fe19c1cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:45:02.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.920+0000 7fe1a43bf640 1 -- 172.21.15.62:0/3821842415 >> 172.21.15.62:0/3821842415 conn(0x7fe19c0aa1d0 msgr2=0x7fe19c0a79d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:45:02.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.920+0000 7fe1a43bf640 1 -- 172.21.15.62:0/3821842415 shutdown_connections 2024-04-19T01:45:02.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:02.920+0000 7fe1a43bf640 1 -- 172.21.15.62:0/3821842415 wait complete. 2024-04-19T01:45:02.934 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-19T01:45:02.934 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-04-19T01:45:02.934 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-19T01:45:03.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f9140619640 10 monclient: get_monmap_and_config 2024-04-19T01:45:03.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f9140619640 10 monclient: build_initial_monmap 2024-04-19T01:45:03.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f9140619640 10 monclient: monmap: 2024-04-19T01:45:03.053 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:45:03.053 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:45:03.053 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:45:03.058547+0000 2024-04-19T01:45:03.053 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:45:03.058547+0000 2024-04-19T01:45:03.053 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:45:03.053 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:45:03.054 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:45:03.054 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:45:03.054 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:45:03.054 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:45:03.054 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:45:03.054 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:45:03.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f9140619640 1 Processor -- start 2024-04-19T01:45:03.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f9140619640 1 -- start start 2024-04-19T01:45:03.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f9140619640 10 monclient: init 2024-04-19T01:45:03.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f9140619640 10 monclient: _reopen_session rank -1 2024-04-19T01:45:03.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f9140619640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-04-19T01:45:03.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f9140619640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f91380a47e0 0x7f913809f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:45:03.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f9140619640 10 monclient(hunting): picked mon.noname-a con 0x7f91380a47e0 addr v2:172.21.15.62:3300/0 2024-04-19T01:45:03.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f9140619640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f91380a4bc0 0x7f91380a4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:45:03.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f9140619640 10 monclient(hunting): picked mon.noname-b con 0x7f91380a4bc0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:45:03.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f9140619640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f91381340a0 0x7f91381344d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:45:03.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f9140619640 10 monclient(hunting): picked mon.noname-c con 0x7f91381340a0 addr v2:172.21.15.71:3300/0 2024-04-19T01:45:03.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f9140619640 10 monclient(hunting): start opening mon connection 2024-04-19T01:45:03.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f9140619640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f913809d4e0 con 0x7f91380a47e0 2024-04-19T01:45:03.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f9140619640 10 monclient(hunting): start opening mon connection 2024-04-19T01:45:03.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f9140619640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f913809e6e0 con 0x7f91381340a0 2024-04-19T01:45:03.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f9140619640 10 monclient(hunting): start opening mon connection 2024-04-19T01:45:03.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f9140619640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f913809e9c0 con 0x7f91380a4bc0 2024-04-19T01:45:03.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f9140619640 10 monclient(hunting): _renew_subs 2024-04-19T01:45:03.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f9140619640 10 monclient(hunting): authenticate will time out at 1088.878190s 2024-04-19T01:45:03.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f913ebb6640 1 -- >> v2:172.21.15.71:3300/0 conn(0x7f91381340a0 msgr2=0x7f91381344d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-04-19T01:45:03.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f913e3b5640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f91380a47e0 0x7f913809f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:45:03.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f913ebb6640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f91381340a0 0x7f91381344d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:45:03.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f913dbb4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f91380a4bc0 0x7f91380a4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:45:03.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f913e3b5640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f91380a47e0 0x7f913809f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45154/0 (socket says 172.21.15.62:45154) 2024-04-19T01:45:03.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f913e3b5640 1 -- 172.21.15.62:0/1326045304 learned_addr learned my addr 172.21.15.62:0/1326045304 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:45:03.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f913e3b5640 10 monclient(hunting): get_auth_request con 0x7f91380a47e0 auth_method 0 2024-04-19T01:45:03.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f913e3b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:45:03.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f913e3b5640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:45:03.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f913e3b5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:45:03.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f913dbb4640 10 monclient(hunting): get_auth_request con 0x7f91380a4bc0 auth_method 0 2024-04-19T01:45:03.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f913dbb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:45:03.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f913dbb4640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:45:03.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f913dbb4640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:45:03.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f913e3b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:45:03.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f913e3b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:45:03.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f913e3b5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:45:03.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f913dbb4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:45:03.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f913dbb4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:45:03.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f913dbb4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:45:03.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f913e3b5640 10 monclient(hunting): handle_auth_done global_id 14238 payload 274 2024-04-19T01:45:03.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f913e3b5640 1 -- 172.21.15.62:0/1326045304 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f91380a4bc0 msgr2=0x7f91380a4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:45:03.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f913e3b5640 1 --2- 172.21.15.62:0/1326045304 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f91380a4bc0 0x7f91380a4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:45:03.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f913e3b5640 1 -- 172.21.15.62:0/1326045304 >> v2:172.21.15.71:3300/0 conn(0x7f91381340a0 msgr2=0x7f91381344d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:45:03.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f913e3b5640 1 --2- 172.21.15.62:0/1326045304 >> v2:172.21.15.71:3300/0 conn(0x7f91381340a0 0x7f91381344d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:45:03.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f913e3b5640 10 monclient: _finish_hunting 0 2024-04-19T01:45:03.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f913e3b5640 1 monclient: found mon.noname-a 2024-04-19T01:45:03.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f913e3b5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:45:03.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f913e3b5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:45:03.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.056+0000 7f913e3b5640 1 -- 172.21.15.62:0/1326045304 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f913809f990 con 0x7f91380a47e0 2024-04-19T01:45:03.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.060+0000 7f913e3b5640 1 -- 172.21.15.62:0/1326045304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91380a47e0 msgr2=0x7f913809f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:45:03.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.060+0000 7f913e3b5640 1 --2- 172.21.15.62:0/1326045304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91380a47e0 0x7f913809f450 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9138069030 tx=0x7f912c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7d404daa1c948a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:45:03.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.060+0000 7f913d3b3640 1 -- 172.21.15.62:0/1326045304 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912c0b0070 con 0x7f91380a47e0 2024-04-19T01:45:03.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.060+0000 7f913d3b3640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:45:03.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.060+0000 7f913d3b3640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:45:03.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.060+0000 7f913d3b3640 10 monclient: dump: 2024-04-19T01:45:03.061 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:45:03.062 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:45:03.062 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:45:03.062 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:45:03.062 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:45:03.062 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:45:03.062 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:45:03.062 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:45:03.062 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:45:03.062 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:45:03.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.060+0000 7f913d3b3640 10 monclient: _finish_auth 0 2024-04-19T01:45:03.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.060+0000 7f913d3b3640 10 monclient: _check_auth_tickets 2024-04-19T01:45:03.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.060+0000 7f913d3b3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:45:03.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.060+0000 7f913d3b3640 1 -- 172.21.15.62:0/1326045304 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f912c0ab600 con 0x7f91380a47e0 2024-04-19T01:45:03.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.060+0000 7f913d3b3640 10 monclient: handle_config config(0 keys) 2024-04-19T01:45:03.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.060+0000 7f913d3b3640 1 -- 172.21.15.62:0/1326045304 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912c0ab9d0 con 0x7f91380a47e0 2024-04-19T01:45:03.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.060+0000 7f9140619640 5 monclient: authenticate success, global_id 14238 2024-04-19T01:45:03.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.060+0000 7f9140619640 10 monclient: get_monmap_and_config success 2024-04-19T01:45:03.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.060+0000 7f9140619640 4 set_mon_vals no callback set 2024-04-19T01:45:03.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.060+0000 7f913d3b3640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:45:03.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.060+0000 7f913d3b3640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:45:03.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.060+0000 7f913d3b3640 10 monclient: dump: 2024-04-19T01:45:03.063 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:45:03.063 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:45:03.063 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:45:03.063 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:45:03.063 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:45:03.063 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:45:03.064 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:45:03.064 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:45:03.064 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:45:03.064 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:45:03.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f9140619640 10 monclient: shutdown 2024-04-19T01:45:03.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f9140619640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:45:03.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f9140619640 1 -- 172.21.15.62:0/1326045304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91380a47e0 msgr2=0x7f913809f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:45:03.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f9140619640 1 --2- 172.21.15.62:0/1326045304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91380a47e0 0x7f913809f450 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9138069030 tx=0x7f912c01c490 comp rx=0 tx=0).stop 2024-04-19T01:45:03.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f9140619640 1 -- 172.21.15.62:0/1326045304 shutdown_connections 2024-04-19T01:45:03.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f9140619640 1 --2- 172.21.15.62:0/1326045304 >> v2:172.21.15.71:3300/0 conn(0x7f91381340a0 0x7f91381344d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:45:03.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f9140619640 1 --2- 172.21.15.62:0/1326045304 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f91380a4bc0 0x7f91380a4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:45:03.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f9140619640 1 --2- 172.21.15.62:0/1326045304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91380a47e0 0x7f913809f450 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:45:03.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f9140619640 1 -- 172.21.15.62:0/1326045304 >> 172.21.15.62:0/1326045304 conn(0x7f91380a23d0 msgr2=0x7f91380a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:45:03.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f9140619640 1 -- 172.21.15.62:0/1326045304 shutdown_connections 2024-04-19T01:45:03.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f9140619640 1 -- 172.21.15.62:0/1326045304 wait complete. 2024-04-19T01:45:03.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f9140619640 10 monclient: build_initial_monmap 2024-04-19T01:45:03.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f9140619640 10 monclient: monmap: 2024-04-19T01:45:03.070 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:45:03.070 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:45:03.070 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:45:03.070 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:45:03.070 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:45:03.070 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:45:03.070 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:45:03.070 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:45:03.070 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:45:03.070 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:45:03.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f9140619640 1 Processor -- start 2024-04-19T01:45:03.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f9140619640 1 -- start start 2024-04-19T01:45:03.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f9140619640 10 monclient: init 2024-04-19T01:45:03.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f9140619640 10 monclient: _reopen_session rank -1 2024-04-19T01:45:03.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f9140619640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-19T01:45:03.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f9140619640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91380a47e0 0x7f913813a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:45:03.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f9140619640 10 monclient(hunting): picked mon.noname-b con 0x7f91380a47e0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:45:03.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f9140619640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91380a4bc0 0x7f91381ccc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:45:03.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f9140619640 10 monclient(hunting): picked mon.noname-a con 0x7f91380a4bc0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:45:03.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f9140619640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f91381340a0 0x7f913811bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:45:03.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f9140619640 10 monclient(hunting): picked mon.noname-c con 0x7f91381340a0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:45:03.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f9140619640 10 monclient(hunting): start opening mon connection 2024-04-19T01:45:03.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f9140619640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f913813df10 con 0x7f91380a4bc0 2024-04-19T01:45:03.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f9140619640 10 monclient(hunting): start opening mon connection 2024-04-19T01:45:03.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f9140619640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f913813dd90 con 0x7f91380a47e0 2024-04-19T01:45:03.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f9140619640 10 monclient(hunting): start opening mon connection 2024-04-19T01:45:03.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f9140619640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f913813e090 con 0x7f91381340a0 2024-04-19T01:45:03.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f9140619640 10 monclient(hunting): _renew_subs 2024-04-19T01:45:03.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f9140619640 10 monclient(hunting): authenticate will time out at 1088.886182s 2024-04-19T01:45:03.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f913e3b5640 1 -- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91380a47e0 msgr2=0x7f913813a2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-04-19T01:45:03.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f913e3b5640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91380a47e0 0x7f913813a2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:45:03.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f913dbb4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91380a4bc0 0x7f91381ccc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:45:03.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f913ebb6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f91381340a0 0x7f913811bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:45:03.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f913ebb6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f91381340a0 0x7f913811bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:58080/0 (socket says 172.21.15.62:58080) 2024-04-19T01:45:03.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f913ebb6640 1 -- 172.21.15.62:0/754328540 learned_addr learned my addr 172.21.15.62:0/754328540 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:45:03.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f913dbb4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91380a4bc0 0x7f91381ccc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45170/0 (socket says 172.21.15.62:45170) 2024-04-19T01:45:03.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f913ebb6640 10 monclient(hunting): get_auth_request con 0x7f91381340a0 auth_method 0 2024-04-19T01:45:03.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f913ebb6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:45:03.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f913ebb6640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:45:03.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f913ebb6640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:45:03.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f913dbb4640 10 monclient(hunting): get_auth_request con 0x7f91380a4bc0 auth_method 0 2024-04-19T01:45:03.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f913dbb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:45:03.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f913dbb4640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:45:03.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f913dbb4640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:45:03.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f913ebb6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:45:03.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f913ebb6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:45:03.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f913ebb6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:45:03.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f913dbb4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:45:03.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f913dbb4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:45:03.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f913dbb4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:45:03.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f913dbb4640 10 monclient(hunting): handle_auth_done global_id 14241 payload 824 2024-04-19T01:45:03.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f913dbb4640 1 -- 172.21.15.62:0/754328540 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f91381340a0 msgr2=0x7f913811bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:45:03.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f913dbb4640 1 --2- 172.21.15.62:0/754328540 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f91381340a0 0x7f913811bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:45:03.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f913dbb4640 1 -- 172.21.15.62:0/754328540 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91380a47e0 msgr2=0x7f913813a2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:45:03.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f913dbb4640 1 --2- 172.21.15.62:0/754328540 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91380a47e0 0x7f913813a2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:45:03.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f913dbb4640 10 monclient: _finish_hunting 0 2024-04-19T01:45:03.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f913dbb4640 1 monclient: found mon.noname-a 2024-04-19T01:45:03.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f913dbb4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:45:03.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f913dbb4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:45:03.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f913dbb4640 1 -- 172.21.15.62:0/754328540 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f913811e280 con 0x7f91380a4bc0 2024-04-19T01:45:03.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.064+0000 7f913ebb6640 1 --2- 172.21.15.62:0/754328540 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f91381340a0 0x7f913811bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:45:03.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f913dbb4640 1 -- 172.21.15.62:0/754328540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91380a4bc0 msgr2=0x7f91381ccc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:45:03.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f913dbb4640 1 --2- 172.21.15.62:0/754328540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91380a4bc0 0x7f91381ccc30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f913400efc0 tx=0x7f913400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:45:03.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f912affd640 1 -- 172.21.15.62:0/754328540 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9134018020 con 0x7f91380a4bc0 2024-04-19T01:45:03.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f912affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:45:03.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f912affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:45:03.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f912affd640 10 monclient: dump: 2024-04-19T01:45:03.075 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:45:03.075 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:45:03.075 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:45:03.075 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:45:03.075 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:45:03.075 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:45:03.075 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:45:03.075 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:45:03.075 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:45:03.075 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:45:03.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f912affd640 10 monclient: _finish_auth 0 2024-04-19T01:45:03.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f912affd640 10 monclient: _check_auth_tickets 2024-04-19T01:45:03.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f912affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:45:03.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f912affd640 1 -- 172.21.15.62:0/754328540 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9134014680 con 0x7f91380a4bc0 2024-04-19T01:45:03.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f912affd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:45:03.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f912affd640 1 -- 172.21.15.62:0/754328540 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9134014a50 con 0x7f91380a4bc0 2024-04-19T01:45:03.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f9140619640 5 monclient: authenticate success, global_id 14241 2024-04-19T01:45:03.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f9140619640 10 monclient: _renew_subs 2024-04-19T01:45:03.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f9140619640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:45:03.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f9140619640 1 -- 172.21.15.62:0/754328540 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91381cc9c0 con 0x7f91380a4bc0 2024-04-19T01:45:03.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f912affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:45:03.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f912affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:45:03.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f912affd640 10 monclient: dump: 2024-04-19T01:45:03.076 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:45:03.076 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:45:03.077 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:45:03.077 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:45:03.077 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:45:03.077 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:45:03.077 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:45:03.077 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:45:03.077 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:45:03.077 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:45:03.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f913cbb2640 4 set_mon_vals no callback set 2024-04-19T01:45:03.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f9140619640 10 monclient: _renew_subs 2024-04-19T01:45:03.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f9140619640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:45:03.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f9140619640 1 -- 172.21.15.62:0/754328540 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f913811eab0 con 0x7f91380a4bc0 2024-04-19T01:45:03.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f9140619640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:45:03.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f9140619640 1 --2- 172.21.15.62:0/754328540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f913811db00 0x7f913811c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:45:03.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f9140619640 10 monclient(hunting): start opening mon connection 2024-04-19T01:45:03.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f9140619640 1 -- 172.21.15.62:0/754328540 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f91380692b0 con 0x7f913811db00 2024-04-19T01:45:03.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f912affd640 1 -- 172.21.15.62:0/754328540 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f913401b020 con 0x7f91380a4bc0 2024-04-19T01:45:03.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f913e3b5640 1 --2- 172.21.15.62:0/754328540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f913811db00 0x7f913811c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:45:03.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f913e3b5640 10 monclient: get_auth_request con 0x7f913811db00 auth_method 0 2024-04-19T01:45:03.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f913e3b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:45:03.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f913e3b5640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:45:03.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f913e3b5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:45:03.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f913e3b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:45:03.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f913e3b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:45:03.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f913e3b5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:45:03.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f912affd640 1 --2- 172.21.15.62:0/754328540 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f911c036420 0x7f911c0388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:45:03.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f912affd640 1 -- 172.21.15.62:0/754328540 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 4738+0+0 (secure 0 0 0) 0x7f913404a0e0 con 0x7f91380a4bc0 2024-04-19T01:45:03.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f913e3b5640 10 monclient(hunting): handle_auth_done global_id 14244 payload 824 2024-04-19T01:45:03.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f913e3b5640 1 -- 172.21.15.62:0/754328540 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f913813b0b0 con 0x7f913811db00 2024-04-19T01:45:03.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f913ebb6640 1 --2- 172.21.15.62:0/754328540 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f911c036420 0x7f911c0388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:45:03.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.068+0000 7f913ebb6640 10 monclient: get_auth_request con 0x7f911c036420 auth_method 0 2024-04-19T01:45:03.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.072+0000 7f913e3b5640 1 -- 172.21.15.62:0/754328540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f913811db00 msgr2=0x7f913811c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:45:03.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.072+0000 7f913e3b5640 1 --2- 172.21.15.62:0/754328540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f913811db00 0x7f913811c460 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f912c0ad000 tx=0x7f912c01acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:45:03.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.072+0000 7f912affd640 1 -- 172.21.15.62:0/754328540 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912c0ba020 con 0x7f913811db00 2024-04-19T01:45:03.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.072+0000 7f912affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:45:03.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.072+0000 7f912affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:45:03.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.072+0000 7f912affd640 10 monclient: dump: 2024-04-19T01:45:03.079 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:45:03.079 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:45:03.079 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:45:03.079 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:45:03.079 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:45:03.079 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:45:03.079 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:45:03.080 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:45:03.080 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:45:03.080 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:45:03.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.072+0000 7f912affd640 1 -- 172.21.15.62:0/754328540 <== mon.0 v2:172.21.15.62:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f912c0b6830 con 0x7f913811db00 2024-04-19T01:45:03.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.072+0000 7f912affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:45:03.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.072+0000 7f912affd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:45:03.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.072+0000 7f912affd640 1 -- 172.21.15.62:0/754328540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f913811db00 msgr2=0x7f913811c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:45:03.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.072+0000 7f912affd640 1 --2- 172.21.15.62:0/754328540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f913811db00 0x7f913811c460 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f912c0ad000 tx=0x7f912c01acd0 comp rx=0 tx=0).stop 2024-04-19T01:45:03.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.072+0000 7f912affd640 1 -- 172.21.15.62:0/754328540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f913811db00 msgr2=0x7f913811c460 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:45:03.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.072+0000 7f912affd640 1 --2- 172.21.15.62:0/754328540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f913811db00 0x7f913811c460 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f912c0ad000 tx=0x7f912c01acd0 comp rx=0 tx=0).stop 2024-04-19T01:45:03.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.072+0000 7f913ebb6640 1 --2- 172.21.15.62:0/754328540 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f911c036420 0x7f911c0388e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9130003fa0 tx=0x7f91300090e0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:45:03.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.076+0000 7f9140619640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:45:03.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.076+0000 7f9140619640 1 --2- 172.21.15.62:0/754328540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91381cd170 0x7f913811c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:45:03.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.076+0000 7f9140619640 10 monclient(hunting): start opening mon connection 2024-04-19T01:45:03.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.076+0000 7f9140619640 1 -- 172.21.15.62:0/754328540 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9138132d00 con 0x7f91381cd170 2024-04-19T01:45:03.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.076+0000 7f913e3b5640 1 --2- 172.21.15.62:0/754328540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91381cd170 0x7f913811c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:45:03.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.076+0000 7f913e3b5640 10 monclient: get_auth_request con 0x7f91381cd170 auth_method 0 2024-04-19T01:45:03.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.076+0000 7f913e3b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:45:03.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.076+0000 7f913e3b5640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:45:03.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.076+0000 7f913e3b5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:45:03.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.076+0000 7f913e3b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:45:03.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.076+0000 7f913e3b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:45:03.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.076+0000 7f913e3b5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:45:03.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.076+0000 7f913e3b5640 10 monclient(hunting): handle_auth_done global_id 14247 payload 824 2024-04-19T01:45:03.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.076+0000 7f913e3b5640 1 -- 172.21.15.62:0/754328540 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f91381dada0 con 0x7f91381cd170 2024-04-19T01:45:03.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.080+0000 7f913e3b5640 1 -- 172.21.15.62:0/754328540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91381cd170 msgr2=0x7f913811c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:45:03.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.080+0000 7f913e3b5640 1 --2- 172.21.15.62:0/754328540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91381cd170 0x7f913811c9a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f912c01ab40 tx=0x7f912c019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:45:03.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.080+0000 7f912affd640 1 -- 172.21.15.62:0/754328540 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912c0ba040 con 0x7f91381cd170 2024-04-19T01:45:03.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.080+0000 7f912affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:45:03.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.080+0000 7f912affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:45:03.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.080+0000 7f912affd640 10 monclient: dump: 2024-04-19T01:45:03.084 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:45:03.084 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:45:03.084 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:45:03.084 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:45:03.084 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:45:03.084 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:45:03.084 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:45:03.084 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:45:03.084 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:45:03.084 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:45:03.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.080+0000 7f912affd640 1 -- 172.21.15.62:0/754328540 <== mon.0 v2:172.21.15.62:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4707 (secure 0 0 0) 0x7f912c004450 con 0x7f91381cd170 2024-04-19T01:45:03.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.080+0000 7f912affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:45:03.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.080+0000 7f912affd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:45:03.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.080+0000 7f912affd640 1 -- 172.21.15.62:0/754328540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91381cd170 msgr2=0x7f913811c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:45:03.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.080+0000 7f912affd640 1 --2- 172.21.15.62:0/754328540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91381cd170 0x7f913811c9a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f912c01ab40 tx=0x7f912c019230 comp rx=0 tx=0).stop 2024-04-19T01:45:03.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.080+0000 7f912affd640 1 -- 172.21.15.62:0/754328540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91381cd170 msgr2=0x7f913811c9a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:45:03.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.080+0000 7f912affd640 1 --2- 172.21.15.62:0/754328540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91381cd170 0x7f913811c9a0 secure :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f912c01ab40 tx=0x7f912c019230 comp rx=0 tx=0).stop 2024-04-19T01:45:03.085 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-19T01:45:03.085 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "a", 2024-04-19T01:45:03.085 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 0, 2024-04-19T01:45:03.085 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "leader", 2024-04-19T01:45:03.085 INFO:teuthology.orchestra.run.smithi062.stdout: "election_epoch": 32, 2024-04-19T01:45:03.086 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 46643, 2024-04-19T01:45:03.086 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum": [ 2024-04-19T01:45:03.086 INFO:teuthology.orchestra.run.smithi062.stdout: 0, 2024-04-19T01:45:03.086 INFO:teuthology.orchestra.run.smithi062.stdout: 2 2024-04-19T01:45:03.086 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:45:03.086 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_age": 0, 2024-04-19T01:45:03.086 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:45:03.086 INFO:teuthology.orchestra.run.smithi062.stdout: "required_con": "2450521708007866372", 2024-04-19T01:45:03.086 INFO:teuthology.orchestra.run.smithi062.stdout: "required_mon": [ 2024-04-19T01:45:03.086 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:45:03.086 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:45:03.086 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:45:03.086 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:45:03.086 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:45:03.087 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:45:03.087 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:45:03.087 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:45:03.087 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:45:03.087 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:45:03.087 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:45:03.087 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:45:03.087 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:45:03.087 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_mon": [ 2024-04-19T01:45:03.087 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:45:03.087 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:45:03.087 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:45:03.087 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:45:03.087 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:45:03.087 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:45:03.088 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:45:03.088 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:45:03.088 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:45:03.088 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:45:03.088 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:45:03.088 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:45:03.088 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:45:03.088 INFO:teuthology.orchestra.run.smithi062.stdout: "outside_quorum": [], 2024-04-19T01:45:03.088 INFO:teuthology.orchestra.run.smithi062.stdout: "extra_probe_peers": [], 2024-04-19T01:45:03.088 INFO:teuthology.orchestra.run.smithi062.stdout: "sync_provider": [], 2024-04-19T01:45:03.088 INFO:teuthology.orchestra.run.smithi062.stdout: "monmap": { 2024-04-19T01:45:03.088 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 1, 2024-04-19T01:45:03.088 INFO:teuthology.orchestra.run.smithi062.stdout: "fsid": "6466e4b6-2520-485f-95b3-9401915fbbad", 2024-04-19T01:45:03.088 INFO:teuthology.orchestra.run.smithi062.stdout: "modified": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:45:03.088 INFO:teuthology.orchestra.run.smithi062.stdout: "created": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:45:03.088 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release": 19, 2024-04-19T01:45:03.089 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release_name": "squid", 2024-04-19T01:45:03.089 INFO:teuthology.orchestra.run.smithi062.stdout: "election_strategy": 1, 2024-04-19T01:45:03.089 INFO:teuthology.orchestra.run.smithi062.stdout: "disallowed_leaders: ": "", 2024-04-19T01:45:03.089 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false, 2024-04-19T01:45:03.089 INFO:teuthology.orchestra.run.smithi062.stdout: "tiebreaker_mon": "", 2024-04-19T01:45:03.089 INFO:teuthology.orchestra.run.smithi062.stdout: "removed_ranks: ": "", 2024-04-19T01:45:03.089 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:45:03.089 INFO:teuthology.orchestra.run.smithi062.stdout: "persistent": [ 2024-04-19T01:45:03.089 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:45:03.089 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:45:03.089 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:45:03.089 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:45:03.089 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:45:03.089 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:45:03.089 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:45:03.090 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:45:03.090 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:45:03.090 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:45:03.090 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:45:03.090 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:45:03.090 INFO:teuthology.orchestra.run.smithi062.stdout: "optional": [] 2024-04-19T01:45:03.090 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:45:03.090 INFO:teuthology.orchestra.run.smithi062.stdout: "mons": [ 2024-04-19T01:45:03.090 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:45:03.090 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 0, 2024-04-19T01:45:03.090 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "a", 2024-04-19T01:45:03.090 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:45:03.090 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:45:03.090 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:45:03.090 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:45:03.091 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3300", 2024-04-19T01:45:03.091 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:45:03.091 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:45:03.091 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:45:03.091 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:45:03.091 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789", 2024-04-19T01:45:03.091 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:45:03.091 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:45:03.091 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:45:03.091 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:45:03.091 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789/0", 2024-04-19T01:45:03.091 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6789/0", 2024-04-19T01:45:03.091 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:45:03.091 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:45:03.092 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:45:03.092 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:45:03.092 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:45:03.092 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 1, 2024-04-19T01:45:03.092 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "b", 2024-04-19T01:45:03.092 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:45:03.092 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:45:03.092 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:45:03.092 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:45:03.092 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:3300", 2024-04-19T01:45:03.092 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:45:03.093 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:45:03.093 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:45:03.093 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:45:03.093 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789", 2024-04-19T01:45:03.093 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:45:03.093 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:45:03.093 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:45:03.093 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:45:03.093 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789/0", 2024-04-19T01:45:03.093 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.71:6789/0", 2024-04-19T01:45:03.093 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:45:03.093 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:45:03.093 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:45:03.094 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:45:03.094 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:45:03.094 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 2, 2024-04-19T01:45:03.094 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "c", 2024-04-19T01:45:03.094 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:45:03.094 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:45:03.094 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:45:03.094 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:45:03.094 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3301", 2024-04-19T01:45:03.094 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:45:03.094 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:45:03.094 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:45:03.094 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:45:03.094 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790", 2024-04-19T01:45:03.095 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:45:03.095 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:45:03.095 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:45:03.095 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:45:03.095 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790/0", 2024-04-19T01:45:03.095 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6790/0", 2024-04-19T01:45:03.095 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:45:03.095 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:45:03.095 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:45:03.095 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:45:03.095 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:45:03.095 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:45:03.095 INFO:teuthology.orchestra.run.smithi062.stdout: "feature_map": { 2024-04-19T01:45:03.095 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": [ 2024-04-19T01:45:03.095 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:45:03.096 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:45:03.096 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:45:03.096 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:45:03.096 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:45:03.096 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:45:03.096 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": [ 2024-04-19T01:45:03.096 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:45:03.096 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:45:03.096 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:45:03.096 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 2 2024-04-19T01:45:03.096 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:45:03.096 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:45:03.096 INFO:teuthology.orchestra.run.smithi062.stdout: "client": [ 2024-04-19T01:45:03.096 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:45:03.096 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:45:03.097 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:45:03.097 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 4 2024-04-19T01:45:03.097 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:45:03.097 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:45:03.097 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:45:03.097 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false 2024-04-19T01:45:03.097 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-19T01:45:03.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.080+0000 7f9140619640 1 -- 172.21.15.62:0/754328540 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f911c036420 msgr2=0x7f911c0388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:45:03.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.080+0000 7f9140619640 1 --2- 172.21.15.62:0/754328540 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f911c036420 0x7f911c0388e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9130003fa0 tx=0x7f91300090e0 comp rx=0 tx=0).stop 2024-04-19T01:45:03.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.080+0000 7f9140619640 10 monclient: shutdown 2024-04-19T01:45:03.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.080+0000 7f9140619640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:45:03.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.080+0000 7f9140619640 1 -- 172.21.15.62:0/754328540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91380a4bc0 msgr2=0x7f91381ccc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:45:03.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.080+0000 7f9140619640 1 --2- 172.21.15.62:0/754328540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91380a4bc0 0x7f91381ccc30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f913400efc0 tx=0x7f913400c490 comp rx=0 tx=0).stop 2024-04-19T01:45:03.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.080+0000 7f913ebb6640 1 -- 172.21.15.62:0/754328540 reap_dead start 2024-04-19T01:45:03.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.080+0000 7f913ebb6640 1 -- 172.21.15.62:0/754328540 reap_dead start 2024-04-19T01:45:03.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.080+0000 7f9140619640 1 -- 172.21.15.62:0/754328540 shutdown_connections 2024-04-19T01:45:03.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.080+0000 7f9140619640 1 -- 172.21.15.62:0/754328540 >> 172.21.15.62:0/754328540 conn(0x7f91380a23d0 msgr2=0x7f91381332a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:45:03.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.080+0000 7f9140619640 1 -- 172.21.15.62:0/754328540 shutdown_connections 2024-04-19T01:45:03.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.080+0000 7f9140619640 1 -- 172.21.15.62:0/754328540 wait complete. 2024-04-19T01:45:03.099 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-19T01:45:03.099 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-19T01:45:03.100 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-19T01:45:03.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f9500b0e640 10 monclient: get_monmap_and_config 2024-04-19T01:45:03.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f9500b0e640 10 monclient: build_initial_monmap 2024-04-19T01:45:03.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f9500b0e640 10 monclient: monmap: 2024-04-19T01:45:03.221 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:45:03.221 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:45:03.221 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:45:03.226289+0000 2024-04-19T01:45:03.221 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:45:03.226289+0000 2024-04-19T01:45:03.221 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:45:03.221 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:45:03.221 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:45:03.221 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:45:03.221 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:45:03.221 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:45:03.222 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:45:03.222 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:45:03.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f9500b0e640 1 Processor -- start 2024-04-19T01:45:03.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f9500b0e640 1 -- start start 2024-04-19T01:45:03.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f9500b0e640 10 monclient: init 2024-04-19T01:45:03.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f9500b0e640 10 monclient: _reopen_session rank -1 2024-04-19T01:45:03.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f9500b0e640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-04-19T01:45:03.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f9500b0e640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f94fc0a47e0 0x7f94fc0a4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:45:03.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f9500b0e640 10 monclient(hunting): picked mon.noname-a con 0x7f94fc0a47e0 addr v2:172.21.15.62:3300/0 2024-04-19T01:45:03.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f9500b0e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f94fc0a5100 0x7f94fc1350b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:45:03.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f9500b0e640 10 monclient(hunting): picked mon.noname-c con 0x7f94fc0a5100 addr v2:172.21.15.71:3300/0 2024-04-19T01:45:03.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f9500b0e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f94fc1355f0 addr v1:172.21.15.71:6789/0 2024-04-19T01:45:03.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f9500b0e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94fc09f7c0 con 0x7f94fc1355f0 2024-04-19T01:45:03.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f9500b0e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:45:03.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f9500b0e640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f94fc09ee90 con 0x7f94fc0a47e0 2024-04-19T01:45:03.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f9500b0e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:45:03.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f9500b0e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f94fc09c2d0 con 0x7f94fc0a5100 2024-04-19T01:45:03.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f9500b0e640 10 monclient(hunting): _renew_subs 2024-04-19T01:45:03.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f9500b0e640 10 monclient(hunting): authenticate will time out at 1089.045754s 2024-04-19T01:45:03.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f94fa59c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f94fc0a47e0 0x7f94fc0a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:45:03.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f94fa59c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f94fc0a47e0 0x7f94fc0a4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45200/0 (socket says 172.21.15.62:45200) 2024-04-19T01:45:03.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f94fa59c640 1 -- 172.21.15.62:0/209665822 learned_addr learned my addr 172.21.15.62:0/209665822 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:45:03.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f94fa59c640 10 monclient(hunting): get_auth_request con 0x7f94fc0a47e0 auth_method 0 2024-04-19T01:45:03.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f94fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:45:03.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f94fa59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:45:03.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f94fad9d640 1 -- 172.21.15.62:0/209665822 >> v1:172.21.15.71:6789/0 conn(0x7f94fc1355f0 legacy=0x7f94fc13c5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.71:6789/0 2024-04-19T01:45:03.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f94fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:45:03.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f94f9d9b640 1 -- 172.21.15.62:0/209665822 >> v2:172.21.15.71:3300/0 conn(0x7f94fc0a5100 msgr2=0x7f94fc1350b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.71:3300/0 2024-04-19T01:45:03.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f94f9d9b640 1 --2- 172.21.15.62:0/209665822 >> v2:172.21.15.71:3300/0 conn(0x7f94fc0a5100 0x7f94fc1350b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:45:03.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f94f959a640 1 -- 172.21.15.62:0/209665822 DispatchQueue::entry inject delay of 0.002000 2024-04-19T01:45:03.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f94fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:45:03.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f94fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:45:03.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f94fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:45:03.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f94fa59c640 10 monclient(hunting): handle_auth_done global_id 14250 payload 274 2024-04-19T01:45:03.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f94fa59c640 1 -- 172.21.15.62:0/209665822 >> v2:172.21.15.71:3300/0 conn(0x7f94fc0a5100 msgr2=0x7f94fc1350b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:45:03.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f94fa59c640 1 --2- 172.21.15.62:0/209665822 >> v2:172.21.15.71:3300/0 conn(0x7f94fc0a5100 0x7f94fc1350b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:45:03.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f94fa59c640 1 -- 172.21.15.62:0/209665822 >> v1:172.21.15.71:6789/0 conn(0x7f94fc1355f0 legacy=0x7f94fc13c5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:45:03.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f94fa59c640 10 monclient: _finish_hunting 0 2024-04-19T01:45:03.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f94fa59c640 1 monclient: found mon.noname-a 2024-04-19T01:45:03.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f94fa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:45:03.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f94fa59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:45:03.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.224+0000 7f94fa59c640 1 -- 172.21.15.62:0/209665822 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94fc09faa0 con 0x7f94fc0a47e0 2024-04-19T01:45:03.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.228+0000 7f94fa59c640 1 -- 172.21.15.62:0/209665822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94fc0a47e0 msgr2=0x7f94fc0a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:45:03.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.228+0000 7f94fa59c640 1 --2- 172.21.15.62:0/209665822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94fc0a47e0 0x7f94fc0a4bc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f94e4009870 tx=0x7f94e401c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ea47447a30bece7 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:45:03.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.228+0000 7f94f959a640 1 -- 172.21.15.62:0/209665822 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e40b0070 con 0x7f94fc0a47e0 2024-04-19T01:45:03.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.228+0000 7f94f959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:45:03.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.228+0000 7f94f959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:45:03.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.228+0000 7f94f959a640 10 monclient: dump: 2024-04-19T01:45:03.229 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:45:03.229 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:45:03.229 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:45:03.229 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:45:03.229 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:45:03.229 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:45:03.229 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:45:03.229 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:45:03.229 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:45:03.229 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:45:03.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.228+0000 7f94f959a640 10 monclient: _finish_auth 0 2024-04-19T01:45:03.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.228+0000 7f94f959a640 10 monclient: _check_auth_tickets 2024-04-19T01:45:03.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.228+0000 7f94f959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:45:03.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.228+0000 7f94f959a640 1 -- 172.21.15.62:0/209665822 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94e40ab600 con 0x7f94fc0a47e0 2024-04-19T01:45:03.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.228+0000 7f9500b0e640 5 monclient: authenticate success, global_id 14250 2024-04-19T01:45:03.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.228+0000 7f9500b0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:45:03.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.228+0000 7f94f959a640 10 monclient: handle_config config(0 keys) 2024-04-19T01:45:03.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.228+0000 7f94f959a640 1 -- 172.21.15.62:0/209665822 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e40ab9d0 con 0x7f94fc0a47e0 2024-04-19T01:45:03.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.228+0000 7f9500b0e640 10 monclient: get_monmap_and_config success 2024-04-19T01:45:03.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.228+0000 7f9500b0e640 4 set_mon_vals no callback set 2024-04-19T01:45:03.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.228+0000 7f94f959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:45:03.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.228+0000 7f94f959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:45:03.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.228+0000 7f94f959a640 10 monclient: dump: 2024-04-19T01:45:03.230 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:45:03.230 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:45:03.230 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:45:03.231 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:45:03.231 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:45:03.231 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:45:03.231 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:45:03.231 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:45:03.231 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:45:03.231 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:45:03.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.228+0000 7f9500b0e640 10 monclient: shutdown 2024-04-19T01:45:03.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.228+0000 7f9500b0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:45:03.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.228+0000 7f9500b0e640 1 -- 172.21.15.62:0/209665822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94fc0a47e0 msgr2=0x7f94fc0a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:45:03.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.228+0000 7f9500b0e640 1 --2- 172.21.15.62:0/209665822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94fc0a47e0 0x7f94fc0a4bc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f94e4009870 tx=0x7f94e401c340 comp rx=0 tx=0).stop 2024-04-19T01:45:03.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.228+0000 7f9500b0e640 1 -- 172.21.15.62:0/209665822 shutdown_connections 2024-04-19T01:45:03.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.228+0000 7f9500b0e640 1 --2- 172.21.15.62:0/209665822 >> v2:172.21.15.71:3300/0 conn(0x7f94fc0a5100 0x7f94fc1350b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:45:03.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.228+0000 7f9500b0e640 1 --2- 172.21.15.62:0/209665822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94fc0a47e0 0x7f94fc0a4bc0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:45:03.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.228+0000 7f9500b0e640 1 -- 172.21.15.62:0/209665822 >> 172.21.15.62:0/209665822 conn(0x7f94fc0a23d0 msgr2=0x7f94fc0a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:45:03.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.228+0000 7f9500b0e640 1 -- 172.21.15.62:0/209665822 shutdown_connections 2024-04-19T01:45:03.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.228+0000 7f9500b0e640 1 -- 172.21.15.62:0/209665822 wait complete. 2024-04-19T01:45:03.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.228+0000 7f9500b0e640 10 monclient: build_initial_monmap 2024-04-19T01:45:03.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.228+0000 7f9500b0e640 10 monclient: monmap: 2024-04-19T01:45:03.232 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:45:03.232 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:45:03.232 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:45:03.232 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:45:03.232 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:45:03.232 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:45:03.232 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:45:03.232 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:45:03.232 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:45:03.232 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:45:03.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f9500b0e640 1 Processor -- start 2024-04-19T01:45:03.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f9500b0e640 1 -- start start 2024-04-19T01:45:03.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f9500b0e640 10 monclient: init 2024-04-19T01:45:03.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f9500b0e640 10 monclient: _reopen_session rank -1 2024-04-19T01:45:03.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f9500b0e640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-19T01:45:03.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f9500b0e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94fc0a47e0 0x7f94fc1cd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:45:03.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f9500b0e640 10 monclient(hunting): picked mon.noname-c con 0x7f94fc0a47e0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:45:03.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f9500b0e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f94fc0a5100 0x7f94fc13c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:45:03.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f9500b0e640 10 monclient(hunting): picked mon.noname-b con 0x7f94fc0a5100 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:45:03.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f9500b0e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94fc1355f0 0x7f94fc13cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:45:03.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f9500b0e640 10 monclient(hunting): picked mon.noname-a con 0x7f94fc1355f0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:45:03.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f9500b0e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:45:03.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f9500b0e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f94fc09f1d0 con 0x7f94fc1355f0 2024-04-19T01:45:03.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f9500b0e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:45:03.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f9500b0e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f94fc09f050 con 0x7f94fc0a5100 2024-04-19T01:45:03.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f9500b0e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:45:03.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f9500b0e640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f94fc09c760 con 0x7f94fc0a47e0 2024-04-19T01:45:03.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f9500b0e640 10 monclient(hunting): _renew_subs 2024-04-19T01:45:03.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f9500b0e640 10 monclient(hunting): authenticate will time out at 1089.052828s 2024-04-19T01:45:03.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94fad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94fc1355f0 0x7f94fc13cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:45:03.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94fad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94fc1355f0 0x7f94fc13cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45212/0 (socket says 172.21.15.62:45212) 2024-04-19T01:45:03.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94fad9d640 1 -- 172.21.15.62:0/3166219492 learned_addr learned my addr 172.21.15.62:0/3166219492 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:45:03.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94fa59c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94fc0a47e0 0x7f94fc1cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:45:03.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94fad9d640 10 monclient(hunting): get_auth_request con 0x7f94fc1355f0 auth_method 0 2024-04-19T01:45:03.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:45:03.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94fad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:45:03.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94fad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:45:03.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94fa59c640 10 monclient(hunting): get_auth_request con 0x7f94fc0a47e0 auth_method 0 2024-04-19T01:45:03.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:45:03.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94fa59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:45:03.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:45:03.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94fad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:45:03.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94fad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:45:03.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94f9d9b640 1 -- 172.21.15.62:0/3166219492 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f94fc0a5100 msgr2=0x7f94fc13c620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2024-04-19T01:45:03.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94fad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:45:03.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94f9d9b640 1 --2- 172.21.15.62:0/3166219492 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f94fc0a5100 0x7f94fc13c620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:45:03.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:45:03.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:45:03.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:45:03.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94fa59c640 10 monclient(hunting): handle_auth_done global_id 4838 payload 824 2024-04-19T01:45:03.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94fa59c640 1 -- 172.21.15.62:0/3166219492 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f94fc0a5100 msgr2=0x7f94fc13c620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:45:03.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94fa59c640 1 --2- 172.21.15.62:0/3166219492 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f94fc0a5100 0x7f94fc13c620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:45:03.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94fa59c640 1 -- 172.21.15.62:0/3166219492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94fc1355f0 msgr2=0x7f94fc13cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:45:03.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94fa59c640 1 --2- 172.21.15.62:0/3166219492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94fc1355f0 0x7f94fc13cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:45:03.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94fa59c640 10 monclient: _finish_hunting 0 2024-04-19T01:45:03.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94fa59c640 1 monclient: found mon.noname-c 2024-04-19T01:45:03.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94fa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:45:03.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94fa59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:45:03.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94fa59c640 1 -- 172.21.15.62:0/3166219492 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94fc139eb0 con 0x7f94fc0a47e0 2024-04-19T01:45:03.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94fad9d640 1 --2- 172.21.15.62:0/3166219492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f94fc1355f0 0x7f94fc13cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:45:03.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94fa59c640 1 -- 172.21.15.62:0/3166219492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94fc0a47e0 msgr2=0x7f94fc1cd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:45:03.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94fa59c640 1 --2- 172.21.15.62:0/3166219492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94fc0a47e0 0x7f94fc1cd700 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f94e40ab000 tx=0x7f94e401ab80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:45:03.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94f2ffd640 1 -- 172.21.15.62:0/3166219492 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e40ba030 con 0x7f94fc0a47e0 2024-04-19T01:45:03.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:45:03.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94f2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:45:03.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94f2ffd640 10 monclient: dump: 2024-04-19T01:45:03.241 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:45:03.241 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:45:03.241 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:45:03.241 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:45:03.241 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:45:03.241 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:45:03.242 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:45:03.242 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:45:03.242 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:45:03.242 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:45:03.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94f2ffd640 10 monclient: _finish_auth 0 2024-04-19T01:45:03.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94f2ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:45:03.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:45:03.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94f2ffd640 1 -- 172.21.15.62:0/3166219492 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94e40ab400 con 0x7f94fc0a47e0 2024-04-19T01:45:03.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f9500b0e640 5 monclient: authenticate success, global_id 4838 2024-04-19T01:45:03.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f9500b0e640 10 monclient: _renew_subs 2024-04-19T01:45:03.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f9500b0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:45:03.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f9500b0e640 1 -- 172.21.15.62:0/3166219492 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94fc06e550 con 0x7f94fc0a47e0 2024-04-19T01:45:03.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94f2ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:45:03.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94f2ffd640 1 -- 172.21.15.62:0/3166219492 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e40b0040 con 0x7f94fc0a47e0 2024-04-19T01:45:03.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:45:03.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:45:03.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94f2ffd640 10 monclient: dump: 2024-04-19T01:45:03.243 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:45:03.243 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:45:03.243 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:45:03.243 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:45:03.243 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:45:03.243 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:45:03.243 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:45:03.243 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:45:03.243 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:45:03.243 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:45:03.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f9500b0e640 10 monclient: _renew_subs 2024-04-19T01:45:03.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f9500b0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:45:03.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f94f8d99640 4 set_mon_vals no callback set 2024-04-19T01:45:03.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.232+0000 7f9500b0e640 1 -- 172.21.15.62:0/3166219492 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f94fc1398f0 con 0x7f94fc0a47e0 2024-04-19T01:45:03.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.236+0000 7f94f2ffd640 1 -- 172.21.15.62:0/3166219492 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f94e40bd070 con 0x7f94fc0a47e0 2024-04-19T01:45:03.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.236+0000 7f9500b0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:45:03.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.236+0000 7f9500b0e640 1 --2- 172.21.15.62:0/3166219492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94fc1361a0 0x7f94fc136580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:45:03.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.236+0000 7f9500b0e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:45:03.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.236+0000 7f9500b0e640 1 -- 172.21.15.62:0/3166219492 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f94fc139b30 con 0x7f94fc1361a0 2024-04-19T01:45:03.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.236+0000 7f94f9d9b640 1 --2- 172.21.15.62:0/3166219492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94fc1361a0 0x7f94fc136580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:45:03.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.236+0000 7f94f9d9b640 10 monclient: get_auth_request con 0x7f94fc1361a0 auth_method 0 2024-04-19T01:45:03.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.236+0000 7f94f9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:45:03.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.236+0000 7f94f9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:45:03.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.236+0000 7f94f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:45:03.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.236+0000 7f94f2ffd640 1 --2- 172.21.15.62:0/3166219492 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f94dc036450 0x7f94dc038910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:45:03.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.236+0000 7f94f2ffd640 1 -- 172.21.15.62:0/3166219492 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(118..118 src has 1..118) ==== 4738+0+0 (secure 0 0 0) 0x7f94e40e35e0 con 0x7f94fc0a47e0 2024-04-19T01:45:03.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.236+0000 7f94f9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:45:03.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.236+0000 7f94f9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:45:03.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.236+0000 7f94f9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:45:03.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.236+0000 7f94f9d9b640 10 monclient(hunting): handle_auth_done global_id 4841 payload 824 2024-04-19T01:45:03.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.236+0000 7f94f9d9b640 1 -- 172.21.15.62:0/3166219492 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f94fc0692b0 con 0x7f94fc1361a0 2024-04-19T01:45:03.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.236+0000 7f94fad9d640 1 --2- 172.21.15.62:0/3166219492 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f94dc036450 0x7f94dc038910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:45:03.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.236+0000 7f94fad9d640 10 monclient: get_auth_request con 0x7f94dc036450 auth_method 0 2024-04-19T01:45:03.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.236+0000 7f94f9d9b640 1 -- 172.21.15.62:0/3166219492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94fc1361a0 msgr2=0x7f94fc136580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:45:03.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.236+0000 7f94f9d9b640 1 --2- 172.21.15.62:0/3166219492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94fc1361a0 0x7f94fc136580 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f94ec003940 tx=0x7f94ec00b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:45:03.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.236+0000 7f94f2ffd640 1 -- 172.21.15.62:0/3166219492 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ec005270 con 0x7f94fc1361a0 2024-04-19T01:45:03.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.236+0000 7f94f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:45:03.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.236+0000 7f94f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:45:03.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.236+0000 7f94f2ffd640 10 monclient: dump: 2024-04-19T01:45:03.245 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:45:03.245 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:45:03.245 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:45:03.245 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:45:03.245 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:45:03.245 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:45:03.246 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:45:03.246 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:45:03.246 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:45:03.246 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:45:03.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.240+0000 7f94f2ffd640 1 -- 172.21.15.62:0/3166219492 <== mon.2 v2:172.21.15.62:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f94ec015070 con 0x7f94fc1361a0 2024-04-19T01:45:03.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.240+0000 7f94f2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:45:03.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.240+0000 7f94f2ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:45:03.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.240+0000 7f94f2ffd640 1 -- 172.21.15.62:0/3166219492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94fc1361a0 msgr2=0x7f94fc136580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:45:03.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.240+0000 7f94f2ffd640 1 --2- 172.21.15.62:0/3166219492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94fc1361a0 0x7f94fc136580 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f94ec003940 tx=0x7f94ec00b8e0 comp rx=0 tx=0).stop 2024-04-19T01:45:03.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.240+0000 7f94f2ffd640 1 -- 172.21.15.62:0/3166219492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94fc1361a0 msgr2=0x7f94fc136580 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:45:03.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.240+0000 7f94f2ffd640 1 --2- 172.21.15.62:0/3166219492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94fc1361a0 0x7f94fc136580 secure :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f94ec003940 tx=0x7f94ec00b8e0 comp rx=0 tx=0).stop 2024-04-19T01:45:03.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.240+0000 7f94fad9d640 1 --2- 172.21.15.62:0/3166219492 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f94dc036450 0x7f94dc038910 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f94fc0728b0 tx=0x7f94f40096c0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:45:03.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.244+0000 7f9500b0e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:45:03.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.244+0000 7f9500b0e640 1 --2- 172.21.15.62:0/3166219492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94fc136ac0 0x7f94fc0701b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:45:03.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.244+0000 7f9500b0e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:45:03.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.244+0000 7f9500b0e640 1 -- 172.21.15.62:0/3166219492 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f94fc1cdc40 con 0x7f94fc136ac0 2024-04-19T01:45:03.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.244+0000 7f94f9d9b640 1 --2- 172.21.15.62:0/3166219492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94fc136ac0 0x7f94fc0701b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:45:03.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.244+0000 7f94f9d9b640 10 monclient: get_auth_request con 0x7f94fc136ac0 auth_method 0 2024-04-19T01:45:03.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.244+0000 7f94f9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:45:03.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.244+0000 7f94f9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:45:03.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.244+0000 7f94f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:45:03.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.244+0000 7f94f9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:45:03.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.244+0000 7f94f9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:45:03.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.244+0000 7f94f9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:45:03.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.244+0000 7f94f9d9b640 10 monclient(hunting): handle_auth_done global_id 4844 payload 824 2024-04-19T01:45:03.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.244+0000 7f94f9d9b640 1 -- 172.21.15.62:0/3166219492 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f94fc1386b0 con 0x7f94fc136ac0 2024-04-19T01:45:03.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.248+0000 7f94f9d9b640 1 -- 172.21.15.62:0/3166219492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94fc136ac0 msgr2=0x7f94fc0701b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:45:03.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.248+0000 7f94f9d9b640 1 --2- 172.21.15.62:0/3166219492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94fc136ac0 0x7f94fc0701b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f94ec003940 tx=0x7f94ec005a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:45:03.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.248+0000 7f94f2ffd640 1 -- 172.21.15.62:0/3166219492 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ec01ae60 con 0x7f94fc136ac0 2024-04-19T01:45:03.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.248+0000 7f94f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:45:03.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.248+0000 7f94f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:45:03.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.248+0000 7f94f2ffd640 10 monclient: dump: 2024-04-19T01:45:03.249 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:45:03.249 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:45:03.249 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:45:03.249 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:45:03.249 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:45:03.249 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:45:03.249 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:45:03.249 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:45:03.250 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:45:03.250 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:45:03.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.248+0000 7f94f2ffd640 1 -- 172.21.15.62:0/3166219492 <== mon.2 v2:172.21.15.62:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f94ec010400 con 0x7f94fc136ac0 2024-04-19T01:45:03.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.248+0000 7f94f2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:45:03.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.248+0000 7f94f2ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:45:03.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.248+0000 7f94f2ffd640 1 -- 172.21.15.62:0/3166219492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94fc136ac0 msgr2=0x7f94fc0701b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:45:03.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.248+0000 7f94f2ffd640 1 --2- 172.21.15.62:0/3166219492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94fc136ac0 0x7f94fc0701b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f94ec003940 tx=0x7f94ec005a30 comp rx=0 tx=0).stop 2024-04-19T01:45:03.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.248+0000 7f94f2ffd640 1 -- 172.21.15.62:0/3166219492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94fc136ac0 msgr2=0x7f94fc0701b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:45:03.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.248+0000 7f94f2ffd640 1 --2- 172.21.15.62:0/3166219492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94fc136ac0 0x7f94fc0701b0 secure :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f94ec003940 tx=0x7f94ec005a30 comp rx=0 tx=0).stop 2024-04-19T01:45:03.250 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-19T01:45:03.250 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "c", 2024-04-19T01:45:03.251 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 2, 2024-04-19T01:45:03.251 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "peon", 2024-04-19T01:45:03.251 INFO:teuthology.orchestra.run.smithi062.stdout: "election_epoch": 32, 2024-04-19T01:45:03.251 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 179454, 2024-04-19T01:45:03.251 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum": [ 2024-04-19T01:45:03.251 INFO:teuthology.orchestra.run.smithi062.stdout: 0, 2024-04-19T01:45:03.251 INFO:teuthology.orchestra.run.smithi062.stdout: 2 2024-04-19T01:45:03.251 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:45:03.251 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_age": 0, 2024-04-19T01:45:03.251 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:45:03.251 INFO:teuthology.orchestra.run.smithi062.stdout: "required_con": "2450521708007866372", 2024-04-19T01:45:03.251 INFO:teuthology.orchestra.run.smithi062.stdout: "required_mon": [ 2024-04-19T01:45:03.251 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:45:03.251 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:45:03.251 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:45:03.251 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:45:03.252 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:45:03.252 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:45:03.252 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:45:03.252 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:45:03.252 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:45:03.252 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:45:03.252 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:45:03.252 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:45:03.252 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:45:03.252 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_mon": [ 2024-04-19T01:45:03.252 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:45:03.252 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:45:03.252 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:45:03.252 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:45:03.252 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:45:03.253 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:45:03.253 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:45:03.253 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:45:03.253 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:45:03.253 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:45:03.253 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:45:03.253 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:45:03.253 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:45:03.253 INFO:teuthology.orchestra.run.smithi062.stdout: "outside_quorum": [], 2024-04-19T01:45:03.253 INFO:teuthology.orchestra.run.smithi062.stdout: "extra_probe_peers": [], 2024-04-19T01:45:03.253 INFO:teuthology.orchestra.run.smithi062.stdout: "sync_provider": [], 2024-04-19T01:45:03.253 INFO:teuthology.orchestra.run.smithi062.stdout: "monmap": { 2024-04-19T01:45:03.253 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 1, 2024-04-19T01:45:03.253 INFO:teuthology.orchestra.run.smithi062.stdout: "fsid": "6466e4b6-2520-485f-95b3-9401915fbbad", 2024-04-19T01:45:03.253 INFO:teuthology.orchestra.run.smithi062.stdout: "modified": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:45:03.253 INFO:teuthology.orchestra.run.smithi062.stdout: "created": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:45:03.254 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release": 19, 2024-04-19T01:45:03.254 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release_name": "squid", 2024-04-19T01:45:03.254 INFO:teuthology.orchestra.run.smithi062.stdout: "election_strategy": 1, 2024-04-19T01:45:03.254 INFO:teuthology.orchestra.run.smithi062.stdout: "disallowed_leaders: ": "", 2024-04-19T01:45:03.254 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false, 2024-04-19T01:45:03.254 INFO:teuthology.orchestra.run.smithi062.stdout: "tiebreaker_mon": "", 2024-04-19T01:45:03.254 INFO:teuthology.orchestra.run.smithi062.stdout: "removed_ranks: ": "", 2024-04-19T01:45:03.254 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:45:03.254 INFO:teuthology.orchestra.run.smithi062.stdout: "persistent": [ 2024-04-19T01:45:03.254 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:45:03.254 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:45:03.255 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:45:03.255 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:45:03.255 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:45:03.255 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:45:03.255 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:45:03.255 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:45:03.255 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:45:03.255 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:45:03.255 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:45:03.255 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:45:03.255 INFO:teuthology.orchestra.run.smithi062.stdout: "optional": [] 2024-04-19T01:45:03.255 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:45:03.255 INFO:teuthology.orchestra.run.smithi062.stdout: "mons": [ 2024-04-19T01:45:03.255 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:45:03.255 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 0, 2024-04-19T01:45:03.256 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "a", 2024-04-19T01:45:03.256 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:45:03.256 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:45:03.256 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:45:03.256 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:45:03.256 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3300", 2024-04-19T01:45:03.256 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:45:03.256 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:45:03.256 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:45:03.256 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:45:03.256 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789", 2024-04-19T01:45:03.257 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:45:03.257 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:45:03.257 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:45:03.257 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:45:03.257 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789/0", 2024-04-19T01:45:03.257 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6789/0", 2024-04-19T01:45:03.257 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:45:03.257 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:45:03.257 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:45:03.257 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:45:03.257 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:45:03.257 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 1, 2024-04-19T01:45:03.257 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "b", 2024-04-19T01:45:03.257 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:45:03.258 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:45:03.258 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:45:03.258 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:45:03.258 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:3300", 2024-04-19T01:45:03.258 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:45:03.258 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:45:03.258 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:45:03.258 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:45:03.258 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789", 2024-04-19T01:45:03.258 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:45:03.258 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:45:03.258 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:45:03.258 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:45:03.258 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789/0", 2024-04-19T01:45:03.258 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.71:6789/0", 2024-04-19T01:45:03.259 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:45:03.259 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:45:03.259 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:45:03.259 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:45:03.259 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:45:03.259 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 2, 2024-04-19T01:45:03.259 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "c", 2024-04-19T01:45:03.259 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:45:03.259 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:45:03.259 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:45:03.259 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:45:03.259 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3301", 2024-04-19T01:45:03.259 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:45:03.259 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:45:03.259 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:45:03.260 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:45:03.260 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790", 2024-04-19T01:45:03.260 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:45:03.260 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:45:03.260 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:45:03.260 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:45:03.260 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790/0", 2024-04-19T01:45:03.260 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6790/0", 2024-04-19T01:45:03.260 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:45:03.260 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:45:03.260 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:45:03.260 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:45:03.260 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:45:03.261 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:45:03.261 INFO:teuthology.orchestra.run.smithi062.stdout: "feature_map": { 2024-04-19T01:45:03.261 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": [ 2024-04-19T01:45:03.261 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:45:03.261 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:45:03.261 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:45:03.261 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:45:03.261 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:45:03.261 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:45:03.261 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": [ 2024-04-19T01:45:03.261 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:45:03.261 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:45:03.261 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:45:03.261 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 4 2024-04-19T01:45:03.261 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:45:03.261 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:45:03.262 INFO:teuthology.orchestra.run.smithi062.stdout: "client": [ 2024-04-19T01:45:03.262 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:45:03.262 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:45:03.262 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:45:03.262 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 5 2024-04-19T01:45:03.262 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:45:03.262 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:45:03.262 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": [ 2024-04-19T01:45:03.262 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:45:03.262 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:45:03.262 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:45:03.262 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:45:03.262 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:45:03.262 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:45:03.263 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:45:03.263 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false 2024-04-19T01:45:03.263 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-19T01:45:03.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.248+0000 7f9500b0e640 1 -- 172.21.15.62:0/3166219492 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f94dc036450 msgr2=0x7f94dc038910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:45:03.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.248+0000 7f9500b0e640 1 --2- 172.21.15.62:0/3166219492 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f94dc036450 0x7f94dc038910 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f94fc0728b0 tx=0x7f94f40096c0 comp rx=0 tx=0).stop 2024-04-19T01:45:03.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.248+0000 7f9500b0e640 10 monclient: shutdown 2024-04-19T01:45:03.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.248+0000 7f9500b0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:45:03.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.248+0000 7f9500b0e640 1 -- 172.21.15.62:0/3166219492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94fc0a47e0 msgr2=0x7f94fc1cd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:45:03.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.248+0000 7f9500b0e640 1 --2- 172.21.15.62:0/3166219492 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f94fc0a47e0 0x7f94fc1cd700 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f94e40ab000 tx=0x7f94e401ab80 comp rx=0 tx=0).stop 2024-04-19T01:45:03.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.248+0000 7f94fad9d640 1 -- 172.21.15.62:0/3166219492 reap_dead start 2024-04-19T01:45:03.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.248+0000 7f94fad9d640 1 -- 172.21.15.62:0/3166219492 reap_dead start 2024-04-19T01:45:03.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.248+0000 7f9500b0e640 1 -- 172.21.15.62:0/3166219492 shutdown_connections 2024-04-19T01:45:03.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.248+0000 7f9500b0e640 1 -- 172.21.15.62:0/3166219492 >> 172.21.15.62:0/3166219492 conn(0x7f94fc0a23d0 msgr2=0x7f94fc0a11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:45:03.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.248+0000 7f9500b0e640 1 -- 172.21.15.62:0/3166219492 shutdown_connections 2024-04-19T01:45:03.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:45:03.248+0000 7f9500b0e640 1 -- 172.21.15.62:0/3166219492 wait complete. 2024-04-19T01:45:03.265 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-19T01:45:03.265 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-04-19T01:45:03.542 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:03.542 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28908]: starting. 2024-04-19T01:45:03.542 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28908]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-28872 2024-04-19T01:45:03.542 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28908]: shutting down. 2024-04-19T01:45:03.546 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:03.546 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:03.546 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:45:03.549 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:45:05.782 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28963]: starting. 2024-04-19T01:45:05.782 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28963]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-28962 2024-04-19T01:45:05.782 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28963]: created object 0... 2024-04-19T01:45:05.782 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28963]: created object 25... 2024-04-19T01:45:05.782 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28963]: created object 49... 2024-04-19T01:45:05.782 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28963]: finishing. 2024-04-19T01:45:05.782 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[28963]: shutting down. 2024-04-19T01:45:06.560 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28964]: starting. 2024-04-19T01:45:06.560 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28964]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-28962 2024-04-19T01:45:06.561 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[28964]: shutting down. 2024-04-19T01:45:08.794 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:08.794 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28997]: starting. 2024-04-19T01:45:08.794 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28997]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-28962 2024-04-19T01:45:08.794 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28997]: created object 0... 2024-04-19T01:45:08.794 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28997]: created object 25... 2024-04-19T01:45:08.794 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28997]: created object 49... 2024-04-19T01:45:08.794 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28997]: finishing. 2024-04-19T01:45:08.794 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[28997]: shutting down. 2024-04-19T01:45:08.861 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:08.861 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28999]: starting. 2024-04-19T01:45:08.862 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28999]: listing objects. 2024-04-19T01:45:08.862 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28999]: listed object 0... 2024-04-19T01:45:08.862 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28999]: listed object 25... 2024-04-19T01:45:08.862 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28999]: saw 50 objects 2024-04-19T01:45:08.862 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[28999]: shutting down. 2024-04-19T01:45:09.602 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:09.602 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28998]: starting. 2024-04-19T01:45:09.602 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28998]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-28962 2024-04-19T01:45:09.602 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[28998]: shutting down. 2024-04-19T01:45:09.605 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:09.605 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:09.605 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:45:09.608 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:45:11.675 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29050]: starting. 2024-04-19T01:45:11.675 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29050]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-29049 2024-04-19T01:45:11.675 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29050]: created object 0... 2024-04-19T01:45:11.675 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29050]: created object 25... 2024-04-19T01:45:11.675 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29050]: created object 49... 2024-04-19T01:45:11.675 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29050]: finishing. 2024-04-19T01:45:11.675 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29050]: shutting down. 2024-04-19T01:45:12.475 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29051]: starting. 2024-04-19T01:45:12.475 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29051]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-29049 2024-04-19T01:45:12.475 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29051]: shutting down. 2024-04-19T01:45:14.685 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:14.685 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29084]: starting. 2024-04-19T01:45:14.685 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29084]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-29049 2024-04-19T01:45:14.685 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29084]: created object 0... 2024-04-19T01:45:14.685 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29084]: created object 25... 2024-04-19T01:45:14.686 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29084]: created object 49... 2024-04-19T01:45:14.686 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29084]: finishing. 2024-04-19T01:45:14.686 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29084]: shutting down. 2024-04-19T01:45:14.749 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:14.750 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29086]: starting. 2024-04-19T01:45:14.750 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29086]: listing objects. 2024-04-19T01:45:14.750 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29086]: listed object 0... 2024-04-19T01:45:14.750 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29086]: listed object 25... 2024-04-19T01:45:14.750 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29086]: saw 50 objects 2024-04-19T01:45:14.750 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29086]: shutting down. 2024-04-19T01:45:15.500 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:15.500 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29085]: starting. 2024-04-19T01:45:15.500 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29085]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-29049 2024-04-19T01:45:15.500 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29085]: shutting down. 2024-04-19T01:45:15.503 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:15.503 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:15.503 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:45:15.507 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:45:18.208 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29137]: starting. 2024-04-19T01:45:18.208 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29137]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-29136 2024-04-19T01:45:18.208 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29137]: created object 0... 2024-04-19T01:45:18.208 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29137]: created object 25... 2024-04-19T01:45:18.208 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29137]: created object 49... 2024-04-19T01:45:18.209 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29137]: finishing. 2024-04-19T01:45:18.209 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29137]: shutting down. 2024-04-19T01:45:18.522 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29138]: starting. 2024-04-19T01:45:18.522 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29138]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-29136 2024-04-19T01:45:18.522 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29138]: shutting down. 2024-04-19T01:45:20.773 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:20.773 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29171]: starting. 2024-04-19T01:45:20.773 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29171]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-29136 2024-04-19T01:45:20.773 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29171]: created object 0... 2024-04-19T01:45:20.773 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29171]: created object 25... 2024-04-19T01:45:20.773 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29171]: created object 49... 2024-04-19T01:45:20.773 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29171]: finishing. 2024-04-19T01:45:20.773 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29171]: shutting down. 2024-04-19T01:45:20.829 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:20.829 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29173]: starting. 2024-04-19T01:45:20.829 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29173]: listing objects. 2024-04-19T01:45:20.829 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29173]: listed object 0... 2024-04-19T01:45:20.829 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29173]: listed object 25... 2024-04-19T01:45:20.829 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29173]: saw 50 objects 2024-04-19T01:45:20.830 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29173]: shutting down. 2024-04-19T01:45:21.465 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:21.465 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29172]: starting. 2024-04-19T01:45:21.465 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29172]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-29136 2024-04-19T01:45:21.465 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29172]: shutting down. 2024-04-19T01:45:21.470 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:21.470 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:21.470 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:45:21.473 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:45:23.715 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29224]: starting. 2024-04-19T01:45:23.715 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29224]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-29223 2024-04-19T01:45:23.715 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29224]: created object 0... 2024-04-19T01:45:23.715 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29224]: created object 25... 2024-04-19T01:45:23.715 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29224]: created object 49... 2024-04-19T01:45:23.715 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29224]: finishing. 2024-04-19T01:45:23.715 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29224]: shutting down. 2024-04-19T01:45:24.485 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29225]: starting. 2024-04-19T01:45:24.486 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29225]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-29223 2024-04-19T01:45:24.486 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29225]: shutting down. 2024-04-19T01:45:26.726 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:26.726 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29258]: starting. 2024-04-19T01:45:26.726 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29258]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-29223 2024-04-19T01:45:26.726 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29258]: created object 0... 2024-04-19T01:45:26.726 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29258]: created object 25... 2024-04-19T01:45:26.726 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29258]: created object 49... 2024-04-19T01:45:26.727 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29258]: finishing. 2024-04-19T01:45:26.727 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29258]: shutting down. 2024-04-19T01:45:26.786 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:26.786 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29260]: starting. 2024-04-19T01:45:26.786 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29260]: listing objects. 2024-04-19T01:45:26.786 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29260]: listed object 0... 2024-04-19T01:45:26.786 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29260]: listed object 25... 2024-04-19T01:45:26.786 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29260]: saw 50 objects 2024-04-19T01:45:26.787 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29260]: shutting down. 2024-04-19T01:45:27.520 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:27.521 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29259]: starting. 2024-04-19T01:45:27.521 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29259]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-29223 2024-04-19T01:45:27.521 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29259]: shutting down. 2024-04-19T01:45:27.526 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:27.526 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:27.526 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:45:27.528 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:45:29.752 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29311]: starting. 2024-04-19T01:45:29.752 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29311]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-29310 2024-04-19T01:45:29.753 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29311]: created object 0... 2024-04-19T01:45:29.753 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29311]: created object 25... 2024-04-19T01:45:29.753 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29311]: created object 49... 2024-04-19T01:45:29.753 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29311]: finishing. 2024-04-19T01:45:29.753 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29311]: shutting down. 2024-04-19T01:45:30.542 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29312]: starting. 2024-04-19T01:45:30.542 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29312]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-29310 2024-04-19T01:45:30.542 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29312]: shutting down. 2024-04-19T01:45:32.693 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:32.693 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29345]: starting. 2024-04-19T01:45:32.693 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29345]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-29310 2024-04-19T01:45:32.693 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29345]: created object 0... 2024-04-19T01:45:32.693 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29345]: created object 25... 2024-04-19T01:45:32.693 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29345]: created object 49... 2024-04-19T01:45:32.693 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29345]: finishing. 2024-04-19T01:45:32.693 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29345]: shutting down. 2024-04-19T01:45:32.749 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:32.749 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29347]: starting. 2024-04-19T01:45:32.749 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29347]: listing objects. 2024-04-19T01:45:32.749 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29347]: listed object 0... 2024-04-19T01:45:32.749 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29347]: listed object 25... 2024-04-19T01:45:32.749 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29347]: saw 50 objects 2024-04-19T01:45:32.749 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29347]: shutting down. 2024-04-19T01:45:33.474 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:33.474 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29346]: starting. 2024-04-19T01:45:33.474 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29346]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-29310 2024-04-19T01:45:33.474 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29346]: shutting down. 2024-04-19T01:45:33.477 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:33.477 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:33.477 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:45:33.480 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:45:35.692 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29398]: starting. 2024-04-19T01:45:35.692 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29398]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-29397 2024-04-19T01:45:35.692 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29398]: created object 0... 2024-04-19T01:45:35.692 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29398]: created object 25... 2024-04-19T01:45:35.692 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29398]: created object 49... 2024-04-19T01:45:35.692 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29398]: finishing. 2024-04-19T01:45:35.693 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29398]: shutting down. 2024-04-19T01:45:36.485 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29399]: starting. 2024-04-19T01:45:36.486 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29399]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-29397 2024-04-19T01:45:36.486 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29399]: shutting down. 2024-04-19T01:45:38.718 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:38.718 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29432]: starting. 2024-04-19T01:45:38.718 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29432]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-29397 2024-04-19T01:45:38.718 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29432]: created object 0... 2024-04-19T01:45:38.718 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29432]: created object 25... 2024-04-19T01:45:38.718 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29432]: created object 49... 2024-04-19T01:45:38.718 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29432]: finishing. 2024-04-19T01:45:38.718 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29432]: shutting down. 2024-04-19T01:45:38.775 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:38.776 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29434]: starting. 2024-04-19T01:45:38.776 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29434]: listing objects. 2024-04-19T01:45:38.776 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29434]: listed object 0... 2024-04-19T01:45:38.776 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29434]: listed object 25... 2024-04-19T01:45:38.776 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29434]: saw 50 objects 2024-04-19T01:45:38.776 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29434]: shutting down. 2024-04-19T01:45:39.520 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:39.520 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29433]: starting. 2024-04-19T01:45:39.520 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29433]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-29397 2024-04-19T01:45:39.520 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29433]: shutting down. 2024-04-19T01:45:39.523 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:39.523 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:39.523 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:45:39.526 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:45:41.706 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29485]: starting. 2024-04-19T01:45:41.707 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29485]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-29484 2024-04-19T01:45:41.707 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29485]: created object 0... 2024-04-19T01:45:41.707 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29485]: created object 25... 2024-04-19T01:45:41.707 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29485]: created object 49... 2024-04-19T01:45:41.707 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29485]: finishing. 2024-04-19T01:45:41.707 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29485]: shutting down. 2024-04-19T01:45:42.486 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29486]: starting. 2024-04-19T01:45:42.486 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29486]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-29484 2024-04-19T01:45:42.487 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29486]: shutting down. 2024-04-19T01:45:44.730 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:44.730 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29519]: starting. 2024-04-19T01:45:44.730 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29519]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-29484 2024-04-19T01:45:44.730 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29519]: created object 0... 2024-04-19T01:45:44.730 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29519]: created object 25... 2024-04-19T01:45:44.730 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29519]: created object 49... 2024-04-19T01:45:44.730 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29519]: finishing. 2024-04-19T01:45:44.730 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29519]: shutting down. 2024-04-19T01:45:45.635 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:45.635 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29521]: starting. 2024-04-19T01:45:45.636 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29521]: listing objects. 2024-04-19T01:45:45.636 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29521]: listed object 0... 2024-04-19T01:45:45.636 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29521]: listed object 25... 2024-04-19T01:45:45.636 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29521]: saw 50 objects 2024-04-19T01:45:45.636 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29521]: shutting down. 2024-04-19T01:45:46.524 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:46.525 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29520]: starting. 2024-04-19T01:45:46.525 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29520]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-29484 2024-04-19T01:45:46.525 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29520]: shutting down. 2024-04-19T01:45:46.528 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:46.528 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:46.528 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:45:46.532 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:45:48.766 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29572]: starting. 2024-04-19T01:45:48.767 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29572]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-29571 2024-04-19T01:45:48.767 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29572]: created object 0... 2024-04-19T01:45:48.767 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29572]: created object 25... 2024-04-19T01:45:48.767 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29572]: created object 49... 2024-04-19T01:45:48.767 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29572]: finishing. 2024-04-19T01:45:48.767 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29572]: shutting down. 2024-04-19T01:45:49.553 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29573]: starting. 2024-04-19T01:45:49.553 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29573]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-29571 2024-04-19T01:45:49.553 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29573]: shutting down. 2024-04-19T01:45:51.694 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:51.694 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29606]: starting. 2024-04-19T01:45:51.694 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29606]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-29571 2024-04-19T01:45:51.694 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29606]: created object 0... 2024-04-19T01:45:51.694 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29606]: created object 25... 2024-04-19T01:45:51.694 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29606]: created object 49... 2024-04-19T01:45:51.694 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29606]: finishing. 2024-04-19T01:45:51.694 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29606]: shutting down. 2024-04-19T01:45:51.749 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:51.749 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29608]: starting. 2024-04-19T01:45:51.749 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29608]: listing objects. 2024-04-19T01:45:51.749 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29608]: listed object 0... 2024-04-19T01:45:51.749 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29608]: listed object 25... 2024-04-19T01:45:51.749 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29608]: saw 50 objects 2024-04-19T01:45:51.749 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29608]: shutting down. 2024-04-19T01:45:52.495 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:52.495 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29607]: starting. 2024-04-19T01:45:52.495 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29607]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-29571 2024-04-19T01:45:52.495 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29607]: shutting down. 2024-04-19T01:45:52.499 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:52.499 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:52.499 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:45:52.501 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:45:54.727 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29659]: starting. 2024-04-19T01:45:54.727 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29659]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-29658 2024-04-19T01:45:54.727 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29659]: created object 0... 2024-04-19T01:45:54.727 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29659]: created object 25... 2024-04-19T01:45:54.727 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29659]: created object 49... 2024-04-19T01:45:54.727 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29659]: finishing. 2024-04-19T01:45:54.727 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29659]: shutting down. 2024-04-19T01:45:55.528 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29660]: starting. 2024-04-19T01:45:55.529 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29660]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-29658 2024-04-19T01:45:55.529 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29660]: shutting down. 2024-04-19T01:45:57.832 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:57.867 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29693]: starting. 2024-04-19T01:45:57.867 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29693]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-29658 2024-04-19T01:45:57.867 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29693]: created object 0... 2024-04-19T01:45:57.868 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29693]: created object 25... 2024-04-19T01:45:57.868 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29693]: created object 49... 2024-04-19T01:45:57.868 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29693]: finishing. 2024-04-19T01:45:57.868 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29693]: shutting down. 2024-04-19T01:45:57.894 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:57.894 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29695]: starting. 2024-04-19T01:45:57.895 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29695]: listing objects. 2024-04-19T01:45:57.895 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29695]: listed object 0... 2024-04-19T01:45:57.895 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29695]: listed object 25... 2024-04-19T01:45:57.895 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29695]: saw 50 objects 2024-04-19T01:45:57.895 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29695]: shutting down. 2024-04-19T01:45:58.642 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:58.642 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29694]: starting. 2024-04-19T01:45:58.642 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29694]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-29658 2024-04-19T01:45:58.642 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29694]: shutting down. 2024-04-19T01:45:58.645 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:58.646 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:45:58.646 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:45:58.649 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:46:01.287 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29746]: starting. 2024-04-19T01:46:01.287 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29746]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-29745 2024-04-19T01:46:01.287 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29746]: created object 0... 2024-04-19T01:46:01.287 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29746]: created object 25... 2024-04-19T01:46:01.287 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29746]: created object 49... 2024-04-19T01:46:01.287 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29746]: finishing. 2024-04-19T01:46:01.287 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29746]: shutting down. 2024-04-19T01:46:01.480 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29747]: starting. 2024-04-19T01:46:01.480 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29747]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-29745 2024-04-19T01:46:01.480 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29747]: shutting down. 2024-04-19T01:46:03.712 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:03.712 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29780]: starting. 2024-04-19T01:46:03.712 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29780]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-29745 2024-04-19T01:46:03.712 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29780]: created object 0... 2024-04-19T01:46:03.712 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29780]: created object 25... 2024-04-19T01:46:03.713 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29780]: created object 49... 2024-04-19T01:46:03.713 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29780]: finishing. 2024-04-19T01:46:03.713 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29780]: shutting down. 2024-04-19T01:46:03.771 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:03.771 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29782]: starting. 2024-04-19T01:46:03.771 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29782]: listing objects. 2024-04-19T01:46:03.771 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29782]: listed object 0... 2024-04-19T01:46:03.771 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29782]: listed object 25... 2024-04-19T01:46:03.771 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29782]: saw 50 objects 2024-04-19T01:46:03.772 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29782]: shutting down. 2024-04-19T01:46:04.503 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:04.503 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29781]: starting. 2024-04-19T01:46:04.503 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29781]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-29745 2024-04-19T01:46:04.503 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29781]: shutting down. 2024-04-19T01:46:04.506 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:04.507 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:04.507 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:46:04.514 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:46:06.756 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29833]: starting. 2024-04-19T01:46:06.756 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29833]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-29832 2024-04-19T01:46:06.756 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29833]: created object 0... 2024-04-19T01:46:06.756 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29833]: created object 25... 2024-04-19T01:46:06.756 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29833]: created object 49... 2024-04-19T01:46:06.756 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29833]: finishing. 2024-04-19T01:46:06.756 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29833]: shutting down. 2024-04-19T01:46:07.537 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29834]: starting. 2024-04-19T01:46:07.537 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29834]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-29832 2024-04-19T01:46:07.537 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29834]: shutting down. 2024-04-19T01:46:09.772 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:09.773 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29867]: starting. 2024-04-19T01:46:09.773 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29867]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-29832 2024-04-19T01:46:09.773 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29867]: created object 0... 2024-04-19T01:46:09.773 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29867]: created object 25... 2024-04-19T01:46:09.773 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29867]: created object 49... 2024-04-19T01:46:09.773 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29867]: finishing. 2024-04-19T01:46:09.773 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29867]: shutting down. 2024-04-19T01:46:09.828 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:09.829 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29869]: starting. 2024-04-19T01:46:09.829 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29869]: listing objects. 2024-04-19T01:46:09.829 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29869]: listed object 0... 2024-04-19T01:46:09.829 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29869]: listed object 25... 2024-04-19T01:46:09.829 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29869]: saw 50 objects 2024-04-19T01:46:09.829 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29869]: shutting down. 2024-04-19T01:46:10.561 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:10.561 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29868]: starting. 2024-04-19T01:46:10.561 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29868]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-29832 2024-04-19T01:46:10.561 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29868]: shutting down. 2024-04-19T01:46:10.564 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:10.564 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:10.565 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:46:10.568 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:46:12.706 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29920]: starting. 2024-04-19T01:46:12.706 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29920]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-29919 2024-04-19T01:46:12.706 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29920]: created object 0... 2024-04-19T01:46:12.706 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29920]: created object 25... 2024-04-19T01:46:12.706 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29920]: created object 49... 2024-04-19T01:46:12.707 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29920]: finishing. 2024-04-19T01:46:12.707 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[29920]: shutting down. 2024-04-19T01:46:13.510 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29921]: starting. 2024-04-19T01:46:13.510 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29921]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-29919 2024-04-19T01:46:13.510 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[29921]: shutting down. 2024-04-19T01:46:15.720 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:15.720 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29954]: starting. 2024-04-19T01:46:15.720 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29954]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-29919 2024-04-19T01:46:15.720 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29954]: created object 0... 2024-04-19T01:46:15.720 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29954]: created object 25... 2024-04-19T01:46:15.720 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29954]: created object 49... 2024-04-19T01:46:15.720 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29954]: finishing. 2024-04-19T01:46:15.720 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[29954]: shutting down. 2024-04-19T01:46:15.780 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:15.780 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29956]: starting. 2024-04-19T01:46:15.780 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29956]: listing objects. 2024-04-19T01:46:15.780 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29956]: listed object 0... 2024-04-19T01:46:15.780 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29956]: listed object 25... 2024-04-19T01:46:15.780 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29956]: saw 50 objects 2024-04-19T01:46:15.781 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[29956]: shutting down. 2024-04-19T01:46:16.509 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:16.510 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29955]: starting. 2024-04-19T01:46:16.510 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29955]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-29919 2024-04-19T01:46:16.510 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[29955]: shutting down. 2024-04-19T01:46:16.513 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:16.513 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:16.513 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:46:16.517 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:46:18.738 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30007]: starting. 2024-04-19T01:46:18.738 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30007]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-30006 2024-04-19T01:46:18.738 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30007]: created object 0... 2024-04-19T01:46:18.738 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30007]: created object 25... 2024-04-19T01:46:18.738 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30007]: created object 49... 2024-04-19T01:46:18.738 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30007]: finishing. 2024-04-19T01:46:18.738 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30007]: shutting down. 2024-04-19T01:46:19.539 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30008]: starting. 2024-04-19T01:46:19.539 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30008]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-30006 2024-04-19T01:46:19.539 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30008]: shutting down. 2024-04-19T01:46:21.911 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:21.911 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30041]: starting. 2024-04-19T01:46:21.911 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30041]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-30006 2024-04-19T01:46:21.912 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30041]: created object 0... 2024-04-19T01:46:21.912 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30041]: created object 25... 2024-04-19T01:46:21.912 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30041]: created object 49... 2024-04-19T01:46:21.912 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30041]: finishing. 2024-04-19T01:46:21.912 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30041]: shutting down. 2024-04-19T01:46:21.972 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:21.972 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30043]: starting. 2024-04-19T01:46:21.972 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30043]: listing objects. 2024-04-19T01:46:21.972 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30043]: listed object 0... 2024-04-19T01:46:21.972 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30043]: listed object 25... 2024-04-19T01:46:21.972 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30043]: saw 50 objects 2024-04-19T01:46:21.972 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30043]: shutting down. 2024-04-19T01:46:22.498 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:22.498 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30042]: starting. 2024-04-19T01:46:22.498 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30042]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-30006 2024-04-19T01:46:22.498 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30042]: shutting down. 2024-04-19T01:46:22.502 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:22.502 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:22.502 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:46:22.506 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:46:24.729 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30094]: starting. 2024-04-19T01:46:24.730 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30094]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-30093 2024-04-19T01:46:24.730 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30094]: created object 0... 2024-04-19T01:46:24.730 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30094]: created object 25... 2024-04-19T01:46:24.730 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30094]: created object 49... 2024-04-19T01:46:24.730 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30094]: finishing. 2024-04-19T01:46:24.730 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30094]: shutting down. 2024-04-19T01:46:25.828 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30095]: starting. 2024-04-19T01:46:25.828 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30095]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-30093 2024-04-19T01:46:25.828 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30095]: shutting down. 2024-04-19T01:46:28.058 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:28.058 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30128]: starting. 2024-04-19T01:46:28.058 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30128]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-30093 2024-04-19T01:46:28.058 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30128]: created object 0... 2024-04-19T01:46:28.058 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30128]: created object 25... 2024-04-19T01:46:28.058 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30128]: created object 49... 2024-04-19T01:46:28.058 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30128]: finishing. 2024-04-19T01:46:28.059 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30128]: shutting down. 2024-04-19T01:46:28.116 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:28.116 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30130]: starting. 2024-04-19T01:46:28.116 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30130]: listing objects. 2024-04-19T01:46:28.116 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30130]: listed object 0... 2024-04-19T01:46:28.116 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30130]: listed object 25... 2024-04-19T01:46:28.117 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30130]: saw 50 objects 2024-04-19T01:46:28.117 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30130]: shutting down. 2024-04-19T01:46:28.862 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:28.862 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30129]: starting. 2024-04-19T01:46:28.863 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30129]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-30093 2024-04-19T01:46:28.863 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30129]: shutting down. 2024-04-19T01:46:28.865 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:28.865 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:28.865 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:46:28.868 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:46:31.104 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30181]: starting. 2024-04-19T01:46:31.104 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30181]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-30180 2024-04-19T01:46:31.104 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30181]: created object 0... 2024-04-19T01:46:31.105 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30181]: created object 25... 2024-04-19T01:46:31.105 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30181]: created object 49... 2024-04-19T01:46:31.105 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30181]: finishing. 2024-04-19T01:46:31.105 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30181]: shutting down. 2024-04-19T01:46:31.492 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30182]: starting. 2024-04-19T01:46:31.492 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30182]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-30180 2024-04-19T01:46:31.492 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30182]: shutting down. 2024-04-19T01:46:33.266 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-19T01:46:33.266 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-04-19T01:46:33.266 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-19T01:46:33.267 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-19T01:46:33.270 INFO:tasks.ceph.mon.b:Started 2024-04-19T01:46:33.270 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-19T01:46:33.270 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:46:33.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d9e292640 10 monclient: get_monmap_and_config 2024-04-19T01:46:33.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d9e292640 10 monclient: build_initial_monmap 2024-04-19T01:46:33.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d9e292640 10 monclient: monmap: 2024-04-19T01:46:33.402 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:46:33.402 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:33.402 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:46:33.407356+0000 2024-04-19T01:46:33.402 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:46:33.407356+0000 2024-04-19T01:46:33.402 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:46:33.402 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:33.402 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:46:33.402 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:46:33.402 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:46:33.402 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:46:33.402 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:46:33.402 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:33.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d9e292640 1 Processor -- start 2024-04-19T01:46:33.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d9e292640 1 -- start start 2024-04-19T01:46:33.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d9e292640 10 monclient: init 2024-04-19T01:46:33.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d9e292640 10 monclient: _reopen_session rank -1 2024-04-19T01:46:33.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d9e292640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-04-19T01:46:33.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d9e292640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7d980a47e0 0x7f7d980a4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:33.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d9e292640 10 monclient(hunting): picked mon.noname-a con 0x7f7d980a47e0 addr v2:172.21.15.62:3300/0 2024-04-19T01:46:33.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d9e292640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f7d980a5100 0x7f7d981350b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:33.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d9e292640 10 monclient(hunting): picked mon.noname-c con 0x7f7d980a5100 addr v2:172.21.15.71:3300/0 2024-04-19T01:46:33.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d9e292640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7d981355f0 addr v1:172.21.15.62:6789/0 2024-04-19T01:46:33.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d9e292640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d9809f7c0 con 0x7f7d981355f0 2024-04-19T01:46:33.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d9e292640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:33.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d9e292640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f7d9809ee90 con 0x7f7d980a47e0 2024-04-19T01:46:33.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d9e292640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:33.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d9e292640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f7d9809c2d0 con 0x7f7d980a5100 2024-04-19T01:46:33.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d9e292640 10 monclient(hunting): _renew_subs 2024-04-19T01:46:33.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d9e292640 10 monclient(hunting): authenticate will time out at 1179.226852s 2024-04-19T01:46:33.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d97fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7d980a47e0 0x7f7d980a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:33.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d97fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7d980a47e0 0x7f7d980a4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53242/0 (socket says 172.21.15.62:53242) 2024-04-19T01:46:33.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d97fff640 1 -- 172.21.15.62:0/2553178158 learned_addr learned my addr 172.21.15.62:0/2553178158 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:46:33.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d97fff640 10 monclient(hunting): get_auth_request con 0x7f7d980a47e0 auth_method 0 2024-04-19T01:46:33.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:33.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d97fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:33.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d97fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:33.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d9c82f640 1 --1- 172.21.15.62:0/2553178158 >> v1:172.21.15.62:6789/0 conn(0x7f7d981355f0 0x7f7d9813c5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:34236/0 (socket says 172.21.15.62:34236) 2024-04-19T01:46:33.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:33.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:33.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:33.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d977fe640 1 --2- 172.21.15.62:0/2553178158 >> v2:172.21.15.71:3300/0 conn(0x7f7d980a5100 0x7f7d981350b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:33.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d977fe640 10 monclient(hunting): get_auth_request con 0x7f7d980a5100 auth_method 0 2024-04-19T01:46:33.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:33.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d977fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:33.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d977fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:33.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d97fff640 10 monclient(hunting): handle_auth_done global_id 14658 payload 274 2024-04-19T01:46:33.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d97fff640 1 -- 172.21.15.62:0/2553178158 >> v2:172.21.15.71:3300/0 conn(0x7f7d980a5100 msgr2=0x7f7d981350b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:46:33.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d97fff640 1 --2- 172.21.15.62:0/2553178158 >> v2:172.21.15.71:3300/0 conn(0x7f7d980a5100 0x7f7d981350b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:33.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d97fff640 1 -- 172.21.15.62:0/2553178158 >> v1:172.21.15.62:6789/0 conn(0x7f7d981355f0 legacy=0x7f7d9813c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:46:33.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d97fff640 10 monclient: _finish_hunting 0 2024-04-19T01:46:33.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d97fff640 1 monclient: found mon.noname-a 2024-04-19T01:46:33.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d97fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:46:33.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d97fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:46:33.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.403+0000 7f7d97fff640 1 -- 172.21.15.62:0/2553178158 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d9809faa0 con 0x7f7d980a47e0 2024-04-19T01:46:33.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d97fff640 1 -- 172.21.15.62:0/2553178158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d980a47e0 msgr2=0x7f7d980a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:33.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d97fff640 1 --2- 172.21.15.62:0/2553178158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d980a47e0 0x7f7d980a4bc0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7d90009e50 tx=0x7f7d9001c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe201e9a64f5bef7 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:33.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d96ffd640 1 -- 172.21.15.62:0/2553178158 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d900a5560 con 0x7f7d980a47e0 2024-04-19T01:46:33.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d96ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:33.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d96ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:46:33.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d96ffd640 10 monclient: dump: 2024-04-19T01:46:33.410 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:33.410 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:33.410 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:33.410 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:33.410 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:33.410 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:33.410 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:33.410 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:33.410 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:33.411 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:33.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d96ffd640 10 monclient: _finish_auth 0 2024-04-19T01:46:33.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d96ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:46:33.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:46:33.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d96ffd640 1 -- 172.21.15.62:0/2553178158 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d900a5700 con 0x7f7d980a47e0 2024-04-19T01:46:33.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d96ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:46:33.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d96ffd640 1 -- 172.21.15.62:0/2553178158 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d900a5ad0 con 0x7f7d980a47e0 2024-04-19T01:46:33.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 5 monclient: authenticate success, global_id 14658 2024-04-19T01:46:33.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 10 monclient: get_monmap_and_config success 2024-04-19T01:46:33.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 4 set_mon_vals no callback set 2024-04-19T01:46:33.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d96ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:33.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d96ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:46:33.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d96ffd640 10 monclient: dump: 2024-04-19T01:46:33.411 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:33.412 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:33.412 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:33.412 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:33.412 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:33.412 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:33.412 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:33.412 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:33.412 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:33.412 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:33.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 10 monclient: shutdown 2024-04-19T01:46:33.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:46:33.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 1 -- 172.21.15.62:0/2553178158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d980a47e0 msgr2=0x7f7d980a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:33.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 1 --2- 172.21.15.62:0/2553178158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d980a47e0 0x7f7d980a4bc0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7d90009e50 tx=0x7f7d9001c7f0 comp rx=0 tx=0).stop 2024-04-19T01:46:33.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 1 -- 172.21.15.62:0/2553178158 shutdown_connections 2024-04-19T01:46:33.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 1 --2- 172.21.15.62:0/2553178158 >> v2:172.21.15.71:3300/0 conn(0x7f7d980a5100 0x7f7d981350b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:33.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 1 --2- 172.21.15.62:0/2553178158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d980a47e0 0x7f7d980a4bc0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:33.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 1 -- 172.21.15.62:0/2553178158 >> 172.21.15.62:0/2553178158 conn(0x7f7d980a23d0 msgr2=0x7f7d980a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:46:33.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 1 -- 172.21.15.62:0/2553178158 shutdown_connections 2024-04-19T01:46:33.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 1 -- 172.21.15.62:0/2553178158 wait complete. 2024-04-19T01:46:33.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 10 monclient: build_initial_monmap 2024-04-19T01:46:33.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 10 monclient: monmap: 2024-04-19T01:46:33.413 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:46:33.413 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:46:33.413 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:46:33.413 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:46:33.413 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:46:33.413 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:33.413 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:46:33.413 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:46:33.413 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:46:33.413 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:33.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 1 Processor -- start 2024-04-19T01:46:33.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 1 -- start start 2024-04-19T01:46:33.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 10 monclient: init 2024-04-19T01:46:33.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 10 monclient: _reopen_session rank -1 2024-04-19T01:46:33.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-19T01:46:33.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7d980a47e0 0x7f7d981cd980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:33.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 10 monclient(hunting): picked mon.noname-b con 0x7f7d980a47e0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:46:33.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d980a5100 0x7f7d9813c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:33.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 10 monclient(hunting): picked mon.noname-a con 0x7f7d980a5100 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:46:33.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7d981355f0 0x7f7d9813cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:33.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 10 monclient(hunting): picked mon.noname-c con 0x7f7d981355f0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:46:33.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:33.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7d9809f1d0 con 0x7f7d980a5100 2024-04-19T01:46:33.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:33.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f7d9809f050 con 0x7f7d980a47e0 2024-04-19T01:46:33.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:33.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f7d9809c760 con 0x7f7d981355f0 2024-04-19T01:46:33.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 10 monclient(hunting): _renew_subs 2024-04-19T01:46:33.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9e292640 10 monclient(hunting): authenticate will time out at 1179.234590s 2024-04-19T01:46:33.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9c82f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7d981355f0 0x7f7d9813cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:33.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d977fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d980a5100 0x7f7d9813c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:33.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9c82f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7d981355f0 0x7f7d9813cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:53664/0 (socket says 172.21.15.62:53664) 2024-04-19T01:46:33.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9c82f640 1 -- 172.21.15.62:0/3595940249 learned_addr learned my addr 172.21.15.62:0/3595940249 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:46:33.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9c82f640 10 monclient(hunting): get_auth_request con 0x7f7d981355f0 auth_method 0 2024-04-19T01:46:33.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9c82f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:33.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9c82f640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:33.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9c82f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:33.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d977fe640 10 monclient(hunting): get_auth_request con 0x7f7d980a5100 auth_method 0 2024-04-19T01:46:33.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:33.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d977fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:33.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d977fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:33.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d97fff640 1 --2- 172.21.15.62:0/3595940249 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7d980a47e0 0x7f7d981cd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:33.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9c82f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:33.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9c82f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:33.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9c82f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:33.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d97fff640 10 monclient(hunting): get_auth_request con 0x7f7d980a47e0 auth_method 0 2024-04-19T01:46:33.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:33.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d97fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:33.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d97fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:33.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:33.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:33.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:33.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d97fff640 1 -- 172.21.15.62:0/3595940249 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7d980a47e0 msgr2=0x7f7d981cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-19T01:46:33.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d97fff640 1 -- 172.21.15.62:0/3595940249 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7d980a47e0 msgr2=0x7f7d981cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:46:33.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d97fff640 1 --2- 172.21.15.62:0/3595940249 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7d980a47e0 0x7f7d981cd980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:46:33.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d97fff640 1 --2- 172.21.15.62:0/3595940249 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7d980a47e0 0x7f7d981cd980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:46:33.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9c82f640 10 monclient(hunting): handle_auth_done global_id 5195 payload 824 2024-04-19T01:46:33.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9c82f640 1 -- 172.21.15.62:0/3595940249 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7d980a47e0 msgr2=0x7f7d981cd980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:46:33.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9c82f640 1 --2- 172.21.15.62:0/3595940249 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7d980a47e0 0x7f7d981cd980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:33.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9c82f640 1 -- 172.21.15.62:0/3595940249 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d980a5100 msgr2=0x7f7d9813c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:33.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9c82f640 1 --2- 172.21.15.62:0/3595940249 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d980a5100 0x7f7d9813c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:33.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9c82f640 10 monclient: _finish_hunting 0 2024-04-19T01:46:33.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9c82f640 1 monclient: found mon.noname-c 2024-04-19T01:46:33.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9c82f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:46:33.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9c82f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:46:33.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d9c82f640 1 -- 172.21.15.62:0/3595940249 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d98139da0 con 0x7f7d981355f0 2024-04-19T01:46:33.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.411+0000 7f7d977fe640 1 --2- 172.21.15.62:0/3595940249 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d980a5100 0x7f7d9813c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:46:33.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.415+0000 7f7d9c82f640 1 -- 172.21.15.62:0/3595940249 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7d981355f0 msgr2=0x7f7d9813cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:33.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.415+0000 7f7d9c82f640 1 --2- 172.21.15.62:0/3595940249 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7d981355f0 0x7f7d9813cb00 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7d8800c410 tx=0x7f7d8800c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:33.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.415+0000 7f7d94ff9640 1 -- 172.21.15.62:0/3595940249 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d88017070 con 0x7f7d981355f0 2024-04-19T01:46:33.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.415+0000 7f7d94ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:33.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.415+0000 7f7d94ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:46:33.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.415+0000 7f7d94ff9640 10 monclient: dump: 2024-04-19T01:46:33.423 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:33.423 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:33.423 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:33.423 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:33.423 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:33.423 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:33.423 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:33.423 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:33.423 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:33.423 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:33.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.415+0000 7f7d94ff9640 10 monclient: _finish_auth 0 2024-04-19T01:46:33.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.415+0000 7f7d94ff9640 10 monclient: _check_auth_tickets 2024-04-19T01:46:33.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.415+0000 7f7d94ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:46:33.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.415+0000 7f7d94ff9640 1 -- 172.21.15.62:0/3595940249 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d880076f0 con 0x7f7d981355f0 2024-04-19T01:46:33.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.415+0000 7f7d9e292640 5 monclient: authenticate success, global_id 5195 2024-04-19T01:46:33.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.415+0000 7f7d9e292640 10 monclient: _renew_subs 2024-04-19T01:46:33.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.415+0000 7f7d9e292640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:46:33.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.415+0000 7f7d9e292640 1 -- 172.21.15.62:0/3595940249 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d9813a080 con 0x7f7d981355f0 2024-04-19T01:46:33.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.415+0000 7f7d94ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T01:46:33.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.415+0000 7f7d94ff9640 1 -- 172.21.15.62:0/3595940249 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d88005a40 con 0x7f7d981355f0 2024-04-19T01:46:33.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.415+0000 7f7d94ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:33.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.415+0000 7f7d94ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:46:33.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.415+0000 7f7d94ff9640 10 monclient: dump: 2024-04-19T01:46:33.424 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:33.424 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:33.424 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:33.425 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:33.425 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:33.425 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:33.425 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:33.425 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:33.425 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:33.425 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:33.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.415+0000 7f7d967fc640 4 set_mon_vals no callback set 2024-04-19T01:46:33.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.415+0000 7f7d9e292640 10 monclient: _renew_subs 2024-04-19T01:46:33.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.415+0000 7f7d9e292640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:46:33.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.415+0000 7f7d9e292640 1 -- 172.21.15.62:0/3595940249 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d9806e590 con 0x7f7d981355f0 2024-04-19T01:46:33.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.415+0000 7f7d9e292640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:33.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.415+0000 7f7d9e292640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:33.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.415+0000 7f7d9e292640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:46:33.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.415+0000 7f7d9e292640 1 -- 172.21.15.62:0/3595940249 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d980692b0 con 0x7f7d981355f0 2024-04-19T01:46:33.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.415+0000 7f7d94ff9640 1 -- 172.21.15.62:0/3595940249 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f7d8801a070 con 0x7f7d981355f0 2024-04-19T01:46:33.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.415+0000 7f7d94ff9640 1 --2- 172.21.15.62:0/3595940249 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f7d78036420 0x7f7d780388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:33.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.415+0000 7f7d94ff9640 1 -- 172.21.15.62:0/3595940249 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(208..208 src has 108..208) ==== 4738+0+0 (secure 0 0 0) 0x7f7d88048960 con 0x7f7d981355f0 2024-04-19T01:46:33.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.415+0000 7f7d97fff640 1 --2- 172.21.15.62:0/3595940249 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f7d78036420 0x7f7d780388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:33.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.415+0000 7f7d97fff640 10 monclient: get_auth_request con 0x7f7d78036420 auth_method 0 2024-04-19T01:46:33.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.419+0000 7f7d97fff640 1 --2- 172.21.15.62:0/3595940249 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f7d78036420 0x7f7d780388e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7d900ab000 tx=0x7f7d900a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:33.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.419+0000 7f7d94ff9640 1 -- 172.21.15.62:0/3595940249 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7d88019e10 con 0x7f7d981355f0 2024-04-19T01:46:33.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.419+0000 7f7d94ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:33.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.419+0000 7f7d94ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:46:33.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.675+0000 7f7d9e292640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:46:33.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.675+0000 7f7d9e292640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:46:33.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.675+0000 7f7d9e292640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:46:33.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.675+0000 7f7d9e292640 1 -- 172.21.15.62:0/3595940249 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7d980a4bc0 con 0x7f7d981355f0 2024-04-19T01:46:33.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.679+0000 7f7d94ff9640 1 -- 172.21.15.62:0/3595940249 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f7d88047380 con 0x7f7d981355f0 2024-04-19T01:46:33.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.679+0000 7f7d94ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:46:33.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.679+0000 7f7d94ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:46:33.677 INFO:teuthology.orchestra.run.smithi062.stdout:{"election_epoch":32,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":91,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","modified":"2024-04-19T01:41:38.497795Z","created":"2024-04-19T01:41:38.497795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:46:33.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.679+0000 7f7d9e292640 1 -- 172.21.15.62:0/3595940249 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f7d78036420 msgr2=0x7f7d780388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:33.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.679+0000 7f7d9e292640 1 --2- 172.21.15.62:0/3595940249 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f7d78036420 0x7f7d780388e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7d900ab000 tx=0x7f7d900a7070 comp rx=0 tx=0).stop 2024-04-19T01:46:33.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.679+0000 7f7d9e292640 10 monclient: shutdown 2024-04-19T01:46:33.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.679+0000 7f7d9e292640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:46:33.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.679+0000 7f7d9e292640 1 -- 172.21.15.62:0/3595940249 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7d981355f0 msgr2=0x7f7d9813cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:33.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.679+0000 7f7d9e292640 1 --2- 172.21.15.62:0/3595940249 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7d981355f0 0x7f7d9813cb00 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7d8800c410 tx=0x7f7d8800c8d0 comp rx=0 tx=0).stop 2024-04-19T01:46:33.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.679+0000 7f7d9e292640 1 -- 172.21.15.62:0/3595940249 shutdown_connections 2024-04-19T01:46:33.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.679+0000 7f7d9e292640 1 --2- 172.21.15.62:0/3595940249 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f7d78036420 0x7f7d780388e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:33.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.679+0000 7f7d9e292640 1 --2- 172.21.15.62:0/3595940249 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f7d981355f0 0x7f7d9813cb00 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:33.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.679+0000 7f7d9e292640 1 --2- 172.21.15.62:0/3595940249 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d980a5100 0x7f7d9813c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:33.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.679+0000 7f7d9e292640 1 --2- 172.21.15.62:0/3595940249 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7d980a47e0 0x7f7d981cd980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:33.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.679+0000 7f7d9e292640 1 -- 172.21.15.62:0/3595940249 >> 172.21.15.62:0/3595940249 conn(0x7f7d980a23d0 msgr2=0x7f7d9806f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:46:33.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.679+0000 7f7d9e292640 1 -- 172.21.15.62:0/3595940249 shutdown_connections 2024-04-19T01:46:33.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:33.679+0000 7f7d9e292640 1 -- 172.21.15.62:0/3595940249 wait complete. 2024-04-19T01:46:33.700 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:33.700 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30215]: starting. 2024-04-19T01:46:33.700 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30215]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-30180 2024-04-19T01:46:33.700 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30215]: created object 0... 2024-04-19T01:46:33.700 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30215]: created object 25... 2024-04-19T01:46:33.700 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30215]: created object 49... 2024-04-19T01:46:33.700 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30215]: finishing. 2024-04-19T01:46:33.700 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30215]: shutting down. 2024-04-19T01:46:33.751 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:33.751 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30217]: starting. 2024-04-19T01:46:33.751 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30217]: listing objects. 2024-04-19T01:46:33.751 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30217]: listed object 0... 2024-04-19T01:46:33.751 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30217]: listed object 25... 2024-04-19T01:46:33.751 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30217]: saw 50 objects 2024-04-19T01:46:33.751 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30217]: shutting down. 2024-04-19T01:46:34.522 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:34.523 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30216]: starting. 2024-04-19T01:46:34.523 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30216]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-30180 2024-04-19T01:46:34.523 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30216]: shutting down. 2024-04-19T01:46:34.527 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:34.527 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:34.527 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:46:34.531 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:46:36.697 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:46:36.741 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30310]: starting. 2024-04-19T01:46:36.742 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30310]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-30309 2024-04-19T01:46:36.742 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30310]: created object 0... 2024-04-19T01:46:36.742 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30310]: created object 25... 2024-04-19T01:46:36.742 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30310]: created object 49... 2024-04-19T01:46:36.742 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30310]: finishing. 2024-04-19T01:46:36.742 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30310]: shutting down. 2024-04-19T01:46:36.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823f205640 10 monclient: get_monmap_and_config 2024-04-19T01:46:36.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823f205640 10 monclient: build_initial_monmap 2024-04-19T01:46:36.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823f205640 10 monclient: monmap: 2024-04-19T01:46:36.830 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:46:36.831 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:36.831 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:46:36.835969+0000 2024-04-19T01:46:36.831 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:46:36.835969+0000 2024-04-19T01:46:36.831 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:46:36.831 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:36.831 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:46:36.832 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:46:36.832 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:46:36.832 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:46:36.832 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:46:36.832 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:36.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823f205640 1 Processor -- start 2024-04-19T01:46:36.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823f205640 1 -- start start 2024-04-19T01:46:36.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823f205640 10 monclient: init 2024-04-19T01:46:36.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823f205640 10 monclient: _reopen_session rank -1 2024-04-19T01:46:36.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823f205640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-04-19T01:46:36.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823f205640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82380a47e0 0x7f823809f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:36.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823f205640 10 monclient(hunting): picked mon.noname-b con 0x7f82380a47e0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:46:36.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823f205640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f82380a4bc0 addr v1:172.21.15.62:6789/0 2024-04-19T01:46:36.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823f205640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8238139560 0x7f823813b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:36.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823f205640 10 monclient(hunting): picked mon.noname-a con 0x7f8238139560 addr v2:172.21.15.62:3300/0 2024-04-19T01:46:36.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823f205640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f823809e6e0 con 0x7f82380a4bc0 2024-04-19T01:46:36.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823f205640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:36.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823f205640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f823809d4e0 con 0x7f8238139560 2024-04-19T01:46:36.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823f205640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:36.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823f205640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f823809e9c0 con 0x7f82380a47e0 2024-04-19T01:46:36.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823f205640 10 monclient(hunting): _renew_subs 2024-04-19T01:46:36.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823f205640 10 monclient(hunting): authenticate will time out at 1182.655706s 2024-04-19T01:46:36.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823cfa1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82380a47e0 0x7f823809f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:36.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823d7a2640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8238139560 0x7f823813b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:36.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823cfa1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82380a47e0 0x7f823809f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:50660/0 (socket says 172.21.15.62:50660) 2024-04-19T01:46:36.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f8237fff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f82380a4bc0 0x7f82380a4fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:60054/0 (socket says 172.21.15.62:60054) 2024-04-19T01:46:36.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823d7a2640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8238139560 0x7f823813b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57136/0 (socket says 172.21.15.62:57136) 2024-04-19T01:46:36.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823cfa1640 1 -- 172.21.15.62:0/1992814145 learned_addr learned my addr 172.21.15.62:0/1992814145 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:46:36.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823cfa1640 10 monclient(hunting): get_auth_request con 0x7f82380a47e0 auth_method 0 2024-04-19T01:46:36.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823cfa1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:36.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823cfa1640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:36.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823cfa1640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:36.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823d7a2640 10 monclient(hunting): get_auth_request con 0x7f8238139560 auth_method 0 2024-04-19T01:46:36.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823d7a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:36.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823d7a2640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:36.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.831+0000 7f823d7a2640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:36.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.835+0000 7f823cfa1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:36.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.835+0000 7f823cfa1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:36.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.835+0000 7f823cfa1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:36.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.835+0000 7f823d7a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:36.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.835+0000 7f823d7a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:36.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.835+0000 7f823d7a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:36.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.835+0000 7f823cfa1640 10 monclient(hunting): handle_auth_done global_id 5210 payload 274 2024-04-19T01:46:36.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.835+0000 7f823cfa1640 1 -- 172.21.15.62:0/1992814145 >> v2:172.21.15.62:3300/0 conn(0x7f8238139560 msgr2=0x7f823813b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:46:36.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.835+0000 7f823cfa1640 1 --2- 172.21.15.62:0/1992814145 >> v2:172.21.15.62:3300/0 conn(0x7f8238139560 0x7f823813b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:36.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.835+0000 7f823cfa1640 1 -- 172.21.15.62:0/1992814145 >> v1:172.21.15.62:6789/0 conn(0x7f82380a4bc0 legacy=0x7f82380a4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:46:36.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.835+0000 7f823cfa1640 10 monclient: _finish_hunting 0 2024-04-19T01:46:36.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.835+0000 7f823cfa1640 1 monclient: found mon.noname-b 2024-04-19T01:46:36.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.835+0000 7f823cfa1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:46:36.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.835+0000 7f823cfa1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:46:36.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.835+0000 7f823cfa1640 1 -- 172.21.15.62:0/1992814145 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f823809f990 con 0x7f82380a47e0 2024-04-19T01:46:36.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.835+0000 7f823d7a2640 1 --2- 172.21.15.62:0/1992814145 >> v2:172.21.15.62:3300/0 conn(0x7f8238139560 0x7f823813b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:46:36.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823cfa1640 1 -- 172.21.15.62:0/1992814145 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82380a47e0 msgr2=0x7f823809f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:36.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823cfa1640 1 --2- 172.21.15.62:0/1992814145 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82380a47e0 0x7f823809f450 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8228009870 tx=0x7f822801c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89a119b0b8b9173c server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:36.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f82377fe640 1 -- 172.21.15.62:0/1992814145 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82280b0070 con 0x7f82380a47e0 2024-04-19T01:46:36.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f82377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:36.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f82377fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:46:36.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f82377fe640 10 monclient: dump: 2024-04-19T01:46:36.843 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:36.843 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:36.843 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:36.843 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:36.843 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:36.843 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:36.843 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:36.843 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:36.843 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:36.843 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:36.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f82377fe640 10 monclient: _finish_auth 0 2024-04-19T01:46:36.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f82377fe640 10 monclient: _check_auth_tickets 2024-04-19T01:46:36.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f82377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:46:36.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f82377fe640 1 -- 172.21.15.62:0/1992814145 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82280ab600 con 0x7f82380a47e0 2024-04-19T01:46:36.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 5 monclient: authenticate success, global_id 5210 2024-04-19T01:46:36.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:46:36.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f82377fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:46:36.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 10 monclient: get_monmap_and_config success 2024-04-19T01:46:36.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f82377fe640 1 -- 172.21.15.62:0/1992814145 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82280ab9d0 con 0x7f82380a47e0 2024-04-19T01:46:36.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 4 set_mon_vals no callback set 2024-04-19T01:46:36.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f82377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:36.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f82377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:46:36.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f82377fe640 10 monclient: dump: 2024-04-19T01:46:36.844 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:36.844 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:36.844 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:36.844 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:36.844 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:36.844 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:36.845 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:36.845 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:36.845 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:36.845 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:36.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 10 monclient: shutdown 2024-04-19T01:46:36.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:46:36.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 1 -- 172.21.15.62:0/1992814145 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82380a47e0 msgr2=0x7f823809f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:36.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 1 --2- 172.21.15.62:0/1992814145 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82380a47e0 0x7f823809f450 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8228009870 tx=0x7f822801c4b0 comp rx=0 tx=0).stop 2024-04-19T01:46:36.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 1 -- 172.21.15.62:0/1992814145 shutdown_connections 2024-04-19T01:46:36.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 1 --2- 172.21.15.62:0/1992814145 >> v2:172.21.15.62:3300/0 conn(0x7f8238139560 0x7f823813b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:36.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 1 --2- 172.21.15.62:0/1992814145 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82380a47e0 0x7f823809f450 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:36.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 1 -- 172.21.15.62:0/1992814145 >> 172.21.15.62:0/1992814145 conn(0x7f82380a23d0 msgr2=0x7f82380a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:46:36.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 1 -- 172.21.15.62:0/1992814145 shutdown_connections 2024-04-19T01:46:36.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 1 -- 172.21.15.62:0/1992814145 wait complete. 2024-04-19T01:46:36.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 10 monclient: build_initial_monmap 2024-04-19T01:46:36.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 10 monclient: monmap: 2024-04-19T01:46:36.845 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:46:36.846 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:46:36.846 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:46:36.846 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:46:36.846 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:46:36.846 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:36.846 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:46:36.846 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:46:36.846 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:46:36.846 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:36.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 1 Processor -- start 2024-04-19T01:46:36.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 1 -- start start 2024-04-19T01:46:36.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 10 monclient: init 2024-04-19T01:46:36.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 10 monclient: _reopen_session rank -1 2024-04-19T01:46:36.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-19T01:46:36.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82380a47e0 0x7f823806d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:36.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 10 monclient(hunting): picked mon.noname-c con 0x7f82380a47e0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:46:36.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82380a4bc0 0x7f823806dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:36.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 10 monclient(hunting): picked mon.noname-b con 0x7f82380a4bc0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:46:36.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8238139560 0x7f823806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:36.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 10 monclient(hunting): picked mon.noname-a con 0x7f8238139560 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:46:36.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:36.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f823809be40 con 0x7f8238139560 2024-04-19T01:46:36.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:36.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f823809bcc0 con 0x7f82380a4bc0 2024-04-19T01:46:36.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:36.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f823809bfc0 con 0x7f82380a47e0 2024-04-19T01:46:36.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 10 monclient(hunting): _renew_subs 2024-04-19T01:46:36.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823f205640 10 monclient(hunting): authenticate will time out at 1182.663051s 2024-04-19T01:46:36.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823cfa1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82380a47e0 0x7f823806d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:36.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823d7a2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8238139560 0x7f823806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:36.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823d7a2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8238139560 0x7f823806e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57146/0 (socket says 172.21.15.62:57146) 2024-04-19T01:46:36.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823cfa1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82380a47e0 0x7f823806d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:50674/0 (socket says 172.21.15.62:50674) 2024-04-19T01:46:36.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823d7a2640 1 -- 172.21.15.62:0/3379951861 learned_addr learned my addr 172.21.15.62:0/3379951861 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:46:36.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823d7a2640 10 monclient(hunting): get_auth_request con 0x7f8238139560 auth_method 0 2024-04-19T01:46:36.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823d7a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:36.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823d7a2640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:36.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823d7a2640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:36.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823cfa1640 10 monclient(hunting): get_auth_request con 0x7f82380a47e0 auth_method 0 2024-04-19T01:46:36.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823cfa1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:36.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823cfa1640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:36.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823cfa1640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:36.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823cfa1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:36.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823cfa1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:36.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823cfa1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823d7a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823d7a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823d7a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823cfa1640 10 monclient(hunting): handle_auth_done global_id 5213 payload 824 2024-04-19T01:46:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823cfa1640 1 -- 172.21.15.62:0/3379951861 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82380a4bc0 msgr2=0x7f823806dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823cfa1640 1 --2- 172.21.15.62:0/3379951861 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82380a4bc0 0x7f823806dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823cfa1640 1 -- 172.21.15.62:0/3379951861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8238139560 msgr2=0x7f823806e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823cfa1640 1 --2- 172.21.15.62:0/3379951861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8238139560 0x7f823806e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823cfa1640 10 monclient: _finish_hunting 0 2024-04-19T01:46:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823cfa1640 1 monclient: found mon.noname-c 2024-04-19T01:46:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823cfa1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:46:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823cfa1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:46:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823cfa1640 1 -- 172.21.15.62:0/3379951861 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82381341c0 con 0x7f82380a47e0 2024-04-19T01:46:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.839+0000 7f823d7a2640 1 --2- 172.21.15.62:0/3379951861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8238139560 0x7f823806e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:46:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.843+0000 7f823cfa1640 1 -- 172.21.15.62:0/3379951861 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82380a47e0 msgr2=0x7f823806d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.843+0000 7f823cfa1640 1 --2- 172.21.15.62:0/3379951861 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82380a47e0 0x7f823806d960 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f823806cc40 tx=0x7f8228019250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.843+0000 7f82357fa640 1 -- 172.21.15.62:0/3379951861 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82280b0050 con 0x7f82380a47e0 2024-04-19T01:46:36.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.843+0000 7f82357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:36.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.843+0000 7f82357fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:46:36.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.843+0000 7f82357fa640 10 monclient: dump: 2024-04-19T01:46:36.850 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:36.850 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:36.850 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:36.850 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:36.850 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:36.850 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:36.850 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:36.850 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:36.850 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:36.850 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:36.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.843+0000 7f82357fa640 10 monclient: _finish_auth 0 2024-04-19T01:46:36.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.843+0000 7f82357fa640 10 monclient: _check_auth_tickets 2024-04-19T01:46:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.843+0000 7f82357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:46:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.843+0000 7f82357fa640 1 -- 172.21.15.62:0/3379951861 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82280b1040 con 0x7f82380a47e0 2024-04-19T01:46:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.843+0000 7f82357fa640 10 monclient: handle_config config(0 keys) 2024-04-19T01:46:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.843+0000 7f82357fa640 1 -- 172.21.15.62:0/3379951861 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82280aac00 con 0x7f82380a47e0 2024-04-19T01:46:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.843+0000 7f82357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.843+0000 7f82357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:46:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.843+0000 7f82357fa640 10 monclient: dump: 2024-04-19T01:46:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:36.852 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:36.852 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:36.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.843+0000 7f823f205640 5 monclient: authenticate success, global_id 5213 2024-04-19T01:46:36.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.843+0000 7f823f205640 10 monclient: _renew_subs 2024-04-19T01:46:36.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.843+0000 7f823f205640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:46:36.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.843+0000 7f823f205640 1 -- 172.21.15.62:0/3379951861 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8238070ea0 con 0x7f82380a47e0 2024-04-19T01:46:36.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.843+0000 7f8236ffd640 4 set_mon_vals no callback set 2024-04-19T01:46:36.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.843+0000 7f823f205640 10 monclient: _renew_subs 2024-04-19T01:46:36.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.843+0000 7f823f205640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:46:36.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.843+0000 7f823f205640 1 -- 172.21.15.62:0/3379951861 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f823806f9f0 con 0x7f82380a47e0 2024-04-19T01:46:36.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.843+0000 7f823f205640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:36.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.843+0000 7f823f205640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:36.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.843+0000 7f823f205640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:46:36.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.843+0000 7f823f205640 1 -- 172.21.15.62:0/3379951861 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82380692b0 con 0x7f82380a47e0 2024-04-19T01:46:36.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.843+0000 7f82357fa640 1 -- 172.21.15.62:0/3379951861 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f82280ab4c0 con 0x7f82380a47e0 2024-04-19T01:46:36.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.843+0000 7f82357fa640 1 --2- 172.21.15.62:0/3379951861 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f8224036420 0x7f82240388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:36.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.843+0000 7f82357fa640 1 -- 172.21.15.62:0/3379951861 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(212..212 src has 108..212) ==== 4738+0+0 (secure 0 0 0) 0x7f82280e3270 con 0x7f82380a47e0 2024-04-19T01:46:36.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.843+0000 7f8237fff640 1 --2- 172.21.15.62:0/3379951861 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f8224036420 0x7f82240388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:36.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.843+0000 7f8237fff640 10 monclient: get_auth_request con 0x7f8224036420 auth_method 0 2024-04-19T01:46:36.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.847+0000 7f8237fff640 1 --2- 172.21.15.62:0/3379951861 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f8224036420 0x7f82240388e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f823806f500 tx=0x7f822c0079f0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:36.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.851+0000 7f82357fa640 1 -- 172.21.15.62:0/3379951861 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f82381341c0 con 0x7f82380a47e0 2024-04-19T01:46:36.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.851+0000 7f82357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:36.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:36.851+0000 7f82357fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:46:37.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:37.075+0000 7f823f205640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:46:37.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:37.075+0000 7f823f205640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:46:37.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:37.075+0000 7f823f205640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:46:37.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:37.075+0000 7f823f205640 1 -- 172.21.15.62:0/3379951861 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8238071b60 con 0x7f82380a47e0 2024-04-19T01:46:37.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:37.075+0000 7f82357fa640 1 -- 172.21.15.62:0/3379951861 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f8228016130 con 0x7f82380a47e0 2024-04-19T01:46:37.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:37.075+0000 7f82357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:46:37.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:37.075+0000 7f82357fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:46:37.074 INFO:teuthology.orchestra.run.smithi062.stdout:{"election_epoch":32,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","modified":"2024-04-19T01:41:38.497795Z","created":"2024-04-19T01:41:38.497795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:46:37.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:37.075+0000 7f823f205640 1 -- 172.21.15.62:0/3379951861 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f8224036420 msgr2=0x7f82240388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:37.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:37.075+0000 7f823f205640 1 --2- 172.21.15.62:0/3379951861 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f8224036420 0x7f82240388e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f823806f500 tx=0x7f822c0079f0 comp rx=0 tx=0).stop 2024-04-19T01:46:37.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:37.075+0000 7f823f205640 10 monclient: shutdown 2024-04-19T01:46:37.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:37.075+0000 7f823f205640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:46:37.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:37.075+0000 7f823f205640 1 -- 172.21.15.62:0/3379951861 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82380a47e0 msgr2=0x7f823806d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:37.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:37.075+0000 7f823f205640 1 --2- 172.21.15.62:0/3379951861 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82380a47e0 0x7f823806d960 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f823806cc40 tx=0x7f8228019250 comp rx=0 tx=0).stop 2024-04-19T01:46:37.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:37.079+0000 7f823f205640 1 -- 172.21.15.62:0/3379951861 shutdown_connections 2024-04-19T01:46:37.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:37.079+0000 7f823f205640 1 --2- 172.21.15.62:0/3379951861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8238139560 0x7f823806e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:37.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:37.079+0000 7f823f205640 1 --2- 172.21.15.62:0/3379951861 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82380a4bc0 0x7f823806dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:37.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:37.079+0000 7f823f205640 1 --2- 172.21.15.62:0/3379951861 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f8224036420 0x7f82240388e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:37.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:37.079+0000 7f823f205640 1 --2- 172.21.15.62:0/3379951861 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f82380a47e0 0x7f823806d960 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:37.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:37.079+0000 7f823f205640 1 -- 172.21.15.62:0/3379951861 >> 172.21.15.62:0/3379951861 conn(0x7f82380a23d0 msgr2=0x7f823812f230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:46:37.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:37.079+0000 7f823f205640 1 -- 172.21.15.62:0/3379951861 shutdown_connections 2024-04-19T01:46:37.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:37.079+0000 7f823f205640 1 -- 172.21.15.62:0/3379951861 wait complete. 2024-04-19T01:46:37.547 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30311]: starting. 2024-04-19T01:46:37.547 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30311]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-30309 2024-04-19T01:46:37.547 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30311]: shutting down. 2024-04-19T01:46:39.773 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:39.773 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30344]: starting. 2024-04-19T01:46:39.774 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30344]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-30309 2024-04-19T01:46:39.774 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30344]: created object 0... 2024-04-19T01:46:39.774 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30344]: created object 25... 2024-04-19T01:46:39.774 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30344]: created object 49... 2024-04-19T01:46:39.774 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30344]: finishing. 2024-04-19T01:46:39.774 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30344]: shutting down. 2024-04-19T01:46:39.987 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:39.987 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30346]: starting. 2024-04-19T01:46:39.987 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30346]: listing objects. 2024-04-19T01:46:39.987 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30346]: listed object 0... 2024-04-19T01:46:39.987 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30346]: listed object 25... 2024-04-19T01:46:39.987 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30346]: saw 50 objects 2024-04-19T01:46:39.987 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30346]: shutting down. 2024-04-19T01:46:40.094 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:46:40.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.227+0000 7f18f8de6640 10 monclient: get_monmap_and_config 2024-04-19T01:46:40.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.227+0000 7f18f8de6640 10 monclient: build_initial_monmap 2024-04-19T01:46:40.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.227+0000 7f18f8de6640 10 monclient: monmap: 2024-04-19T01:46:40.226 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:46:40.226 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:40.226 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:46:40.232028+0000 2024-04-19T01:46:40.226 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:46:40.232028+0000 2024-04-19T01:46:40.226 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:46:40.227 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:40.227 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:46:40.227 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:46:40.227 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:46:40.227 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:46:40.227 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:46:40.227 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:40.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.227+0000 7f18f8de6640 1 Processor -- start 2024-04-19T01:46:40.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.227+0000 7f18f8de6640 1 -- start start 2024-04-19T01:46:40.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.227+0000 7f18f8de6640 10 monclient: init 2024-04-19T01:46:40.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.227+0000 7f18f8de6640 10 monclient: _reopen_session rank -1 2024-04-19T01:46:40.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.227+0000 7f18f8de6640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-04-19T01:46:40.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.227+0000 7f18f8de6640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f18f41378d0 0x7f18f4137320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:40.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.227+0000 7f18f8de6640 10 monclient(hunting): picked mon.noname-c con 0x7f18f41378d0 addr v2:172.21.15.71:3300/0 2024-04-19T01:46:40.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.227+0000 7f18f8de6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f18f4137cb0 addr v1:172.21.15.62:6789/0 2024-04-19T01:46:40.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.227+0000 7f18f8de6640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f18f40a2be0 0x7f18f413f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:40.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.227+0000 7f18f8de6640 10 monclient(hunting): picked mon.noname-a con 0x7f18f40a2be0 addr v2:172.21.15.62:3300/0 2024-04-19T01:46:40.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.227+0000 7f18f8de6640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18f4072cb0 con 0x7f18f4137cb0 2024-04-19T01:46:40.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.227+0000 7f18f8de6640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:40.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.227+0000 7f18f8de6640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f18f4075f40 con 0x7f18f40a2be0 2024-04-19T01:46:40.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.227+0000 7f18f8de6640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:40.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.227+0000 7f18f8de6640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f18f4075ab0 con 0x7f18f41378d0 2024-04-19T01:46:40.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.227+0000 7f18f8de6640 10 monclient(hunting): _renew_subs 2024-04-19T01:46:40.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.227+0000 7f18f8de6640 10 monclient(hunting): authenticate will time out at 1186.051992s 2024-04-19T01:46:40.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.231+0000 7f18f1d9b640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f18f4137cb0 0x7f18f40a24d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:60068/0 (socket says 172.21.15.62:60068) 2024-04-19T01:46:40.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.231+0000 7f18f1d9b640 1 -- 172.21.15.62:0/3389578859 learned_addr learned my addr 172.21.15.62:0/3389578859 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:46:40.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.231+0000 7f18f2d9d640 1 --2- 172.21.15.62:0/3389578859 >> v2:172.21.15.62:3300/0 conn(0x7f18f40a2be0 0x7f18f413f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:40.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.231+0000 7f18f2d9d640 10 monclient(hunting): get_auth_request con 0x7f18f40a2be0 auth_method 0 2024-04-19T01:46:40.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.231+0000 7f18f2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:40.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.231+0000 7f18f2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:40.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.231+0000 7f18f2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:40.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.231+0000 7f18f259c640 1 --2- 172.21.15.62:0/3389578859 >> v2:172.21.15.71:3300/0 conn(0x7f18f41378d0 0x7f18f4137320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:40.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.231+0000 7f18f259c640 10 monclient(hunting): get_auth_request con 0x7f18f41378d0 auth_method 0 2024-04-19T01:46:40.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.231+0000 7f18f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:40.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.231+0000 7f18f259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:40.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.231+0000 7f18f259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:40.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.231+0000 7f18f259c640 1 -- 172.21.15.62:0/3389578859 >> v2:172.21.15.71:3300/0 conn(0x7f18f41378d0 msgr2=0x7f18f4137320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:46:40.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.231+0000 7f18f259c640 1 -- 172.21.15.62:0/3389578859 >> v2:172.21.15.71:3300/0 conn(0x7f18f41378d0 msgr2=0x7f18f4137320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:46:40.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.231+0000 7f18f259c640 1 --2- 172.21.15.62:0/3389578859 >> v2:172.21.15.71:3300/0 conn(0x7f18f41378d0 0x7f18f4137320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:46:40.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.231+0000 7f18f259c640 1 --2- 172.21.15.62:0/3389578859 >> v2:172.21.15.71:3300/0 conn(0x7f18f41378d0 0x7f18f4137320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:46:40.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.231+0000 7f18f1d9b640 1 -- 172.21.15.62:0/3389578859 >> v1:172.21.15.62:6789/0 conn(0x7f18f4137cb0 legacy=0x7f18f40a24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:40.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.231+0000 7f18f2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:40.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.231+0000 7f18f2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:40.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.231+0000 7f18f2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:40.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.231+0000 7f18f2d9d640 10 monclient(hunting): handle_auth_done global_id 14697 payload 274 2024-04-19T01:46:40.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.231+0000 7f18f2d9d640 1 -- 172.21.15.62:0/3389578859 >> v2:172.21.15.71:3300/0 conn(0x7f18f41378d0 msgr2=0x7f18f4137320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:46:40.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.231+0000 7f18f2d9d640 1 --2- 172.21.15.62:0/3389578859 >> v2:172.21.15.71:3300/0 conn(0x7f18f41378d0 0x7f18f4137320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:40.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.231+0000 7f18f2d9d640 1 -- 172.21.15.62:0/3389578859 >> v1:172.21.15.62:6789/0 conn(0x7f18f4137cb0 legacy=0x7f18f40a24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:40.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.231+0000 7f18f2d9d640 10 monclient: _finish_hunting 0 2024-04-19T01:46:40.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.231+0000 7f18f2d9d640 1 monclient: found mon.noname-a 2024-04-19T01:46:40.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.231+0000 7f18f2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:46:40.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.231+0000 7f18f2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:46:40.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.231+0000 7f18f2d9d640 1 -- 172.21.15.62:0/3389578859 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18f40a07d0 con 0x7f18f40a2be0 2024-04-19T01:46:40.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.235+0000 7f18f2d9d640 1 -- 172.21.15.62:0/3389578859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f40a2be0 msgr2=0x7f18f413f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:40.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.235+0000 7f18f2d9d640 1 --2- 172.21.15.62:0/3389578859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f40a2be0 0x7f18f413f660 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f18f406dc70 tx=0x7f18ec01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c1a4a6193acd70 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:40.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.235+0000 7f18f159a640 1 -- 172.21.15.62:0/3389578859 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18ec0b0070 con 0x7f18f40a2be0 2024-04-19T01:46:40.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.235+0000 7f18f159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:40.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.235+0000 7f18f159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:46:40.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.235+0000 7f18f159a640 10 monclient: dump: 2024-04-19T01:46:40.236 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:40.236 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:40.236 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:40.236 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:40.236 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:40.236 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:40.236 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:40.236 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:40.236 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:40.236 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:40.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.235+0000 7f18f159a640 10 monclient: _finish_auth 0 2024-04-19T01:46:40.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.235+0000 7f18f159a640 10 monclient: _check_auth_tickets 2024-04-19T01:46:40.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.235+0000 7f18f159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:46:40.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.235+0000 7f18f159a640 1 -- 172.21.15.62:0/3389578859 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18ec0a4e50 con 0x7f18f40a2be0 2024-04-19T01:46:40.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.235+0000 7f18f159a640 10 monclient: handle_config config(0 keys) 2024-04-19T01:46:40.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.235+0000 7f18f159a640 1 -- 172.21.15.62:0/3389578859 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18ec0ab820 con 0x7f18f40a2be0 2024-04-19T01:46:40.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.235+0000 7f18f8de6640 5 monclient: authenticate success, global_id 14697 2024-04-19T01:46:40.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.235+0000 7f18f8de6640 10 monclient: get_monmap_and_config success 2024-04-19T01:46:40.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.235+0000 7f18f8de6640 4 set_mon_vals no callback set 2024-04-19T01:46:40.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.235+0000 7f18f159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:40.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.235+0000 7f18f159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:46:40.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.235+0000 7f18f159a640 10 monclient: dump: 2024-04-19T01:46:40.237 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:40.237 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:40.237 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:40.237 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:40.237 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:40.237 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:40.237 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:40.238 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:40.238 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:40.238 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:40.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.235+0000 7f18f8de6640 10 monclient: shutdown 2024-04-19T01:46:40.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.235+0000 7f18f8de6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:46:40.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.235+0000 7f18f8de6640 1 -- 172.21.15.62:0/3389578859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f40a2be0 msgr2=0x7f18f413f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:40.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.235+0000 7f18f8de6640 1 --2- 172.21.15.62:0/3389578859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f40a2be0 0x7f18f413f660 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f18f406dc70 tx=0x7f18ec01c3c0 comp rx=0 tx=0).stop 2024-04-19T01:46:40.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f8de6640 1 -- 172.21.15.62:0/3389578859 shutdown_connections 2024-04-19T01:46:40.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f8de6640 1 --2- 172.21.15.62:0/3389578859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f40a2be0 0x7f18f413f660 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:40.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f8de6640 1 --2- 172.21.15.62:0/3389578859 >> v2:172.21.15.71:3300/0 conn(0x7f18f41378d0 0x7f18f4137320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:40.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f8de6640 1 -- 172.21.15.62:0/3389578859 >> 172.21.15.62:0/3389578859 conn(0x7f18f40a35c0 msgr2=0x7f18f40a39d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:46:40.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f8de6640 1 -- 172.21.15.62:0/3389578859 shutdown_connections 2024-04-19T01:46:40.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f8de6640 1 -- 172.21.15.62:0/3389578859 wait complete. 2024-04-19T01:46:40.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f8de6640 10 monclient: build_initial_monmap 2024-04-19T01:46:40.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f8de6640 10 monclient: monmap: 2024-04-19T01:46:40.242 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:46:40.242 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:46:40.242 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:46:40.242 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:46:40.242 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:46:40.242 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:40.242 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:46:40.242 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:46:40.242 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:46:40.243 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:40.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f8de6640 1 Processor -- start 2024-04-19T01:46:40.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f8de6640 1 -- start start 2024-04-19T01:46:40.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f8de6640 10 monclient: init 2024-04-19T01:46:40.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f8de6640 10 monclient: _reopen_session rank -1 2024-04-19T01:46:40.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f8de6640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-19T01:46:40.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f8de6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f40a2be0 0x7f18f41cf850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:40.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f8de6640 10 monclient(hunting): picked mon.noname-a con 0x7f18f40a2be0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:46:40.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f8de6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18f41378d0 0x7f18f4136430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:40.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f8de6640 10 monclient(hunting): picked mon.noname-b con 0x7f18f41378d0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:46:40.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f8de6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f18f4137cb0 0x7f18f411d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:40.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f8de6640 10 monclient(hunting): picked mon.noname-c con 0x7f18f4137cb0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:46:40.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f8de6640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:40.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f8de6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f18f4142750 con 0x7f18f40a2be0 2024-04-19T01:46:40.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f8de6640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:40.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f8de6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f18f41425d0 con 0x7f18f41378d0 2024-04-19T01:46:40.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f8de6640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:40.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f8de6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f18f41428d0 con 0x7f18f4137cb0 2024-04-19T01:46:40.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f8de6640 10 monclient(hunting): _renew_subs 2024-04-19T01:46:40.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f8de6640 10 monclient(hunting): authenticate will time out at 1186.061720s 2024-04-19T01:46:40.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f259c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f40a2be0 0x7f18f41cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:40.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f259c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f40a2be0 0x7f18f41cf850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57160/0 (socket says 172.21.15.62:57160) 2024-04-19T01:46:40.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f259c640 1 -- 172.21.15.62:0/3946531235 learned_addr learned my addr 172.21.15.62:0/3946531235 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:46:40.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f259c640 10 monclient(hunting): get_auth_request con 0x7f18f40a2be0 auth_method 0 2024-04-19T01:46:40.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:40.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:40.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:40.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f2d9d640 1 --2- 172.21.15.62:0/3946531235 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f18f4137cb0 0x7f18f411d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:40.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f2d9d640 10 monclient(hunting): get_auth_request con 0x7f18f4137cb0 auth_method 0 2024-04-19T01:46:40.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:40.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:40.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:40.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:40.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:40.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:40.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:40.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:40.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:40.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f1d9b640 1 --2- 172.21.15.62:0/3946531235 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18f41378d0 0x7f18f4136430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:40.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f259c640 10 monclient(hunting): handle_auth_done global_id 14703 payload 824 2024-04-19T01:46:40.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f259c640 1 -- 172.21.15.62:0/3946531235 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f18f4137cb0 msgr2=0x7f18f411d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:40.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f259c640 1 --2- 172.21.15.62:0/3946531235 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f18f4137cb0 0x7f18f411d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:40.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f259c640 1 -- 172.21.15.62:0/3946531235 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18f41378d0 msgr2=0x7f18f4136430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:40.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f259c640 1 --2- 172.21.15.62:0/3946531235 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18f41378d0 0x7f18f4136430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:40.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f259c640 10 monclient: _finish_hunting 0 2024-04-19T01:46:40.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f259c640 1 monclient: found mon.noname-a 2024-04-19T01:46:40.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:46:40.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:46:40.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f259c640 1 -- 172.21.15.62:0/3946531235 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18f4120de0 con 0x7f18f40a2be0 2024-04-19T01:46:40.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f1d9b640 10 monclient: get_auth_request con 0x7f18f41378d0 auth_method 0 2024-04-19T01:46:40.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.239+0000 7f18f1d9b640 1 --2- 172.21.15.62:0/3946531235 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18f41378d0 0x7f18f4136430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:46:40.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.243+0000 7f18f259c640 1 -- 172.21.15.62:0/3946531235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f40a2be0 msgr2=0x7f18f41cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:40.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.243+0000 7f18f259c640 1 --2- 172.21.15.62:0/3946531235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f40a2be0 0x7f18f41cf850 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f18dc00e810 tx=0x7f18dc00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:40.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.243+0000 7f18eaffd640 1 -- 172.21.15.62:0/3946531235 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18dc00f040 con 0x7f18f40a2be0 2024-04-19T01:46:40.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.243+0000 7f18eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:40.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.243+0000 7f18eaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:46:40.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.243+0000 7f18eaffd640 10 monclient: dump: 2024-04-19T01:46:40.246 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:40.246 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:40.246 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:40.246 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:40.246 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:40.246 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:40.246 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:40.246 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:40.246 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:40.247 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:40.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.243+0000 7f18eaffd640 10 monclient: _finish_auth 0 2024-04-19T01:46:40.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.243+0000 7f18eaffd640 10 monclient: _check_auth_tickets 2024-04-19T01:46:40.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.243+0000 7f18eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:46:40.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.243+0000 7f18f8de6640 5 monclient: authenticate success, global_id 14703 2024-04-19T01:46:40.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.243+0000 7f18eaffd640 1 -- 172.21.15.62:0/3946531235 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18dc0098a0 con 0x7f18f40a2be0 2024-04-19T01:46:40.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.243+0000 7f18eaffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:46:40.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.243+0000 7f18eaffd640 1 -- 172.21.15.62:0/3946531235 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18dc009c70 con 0x7f18f40a2be0 2024-04-19T01:46:40.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.243+0000 7f18eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:40.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.243+0000 7f18eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:46:40.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.243+0000 7f18f0d99640 4 set_mon_vals no callback set 2024-04-19T01:46:40.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.243+0000 7f18eaffd640 10 monclient: dump: 2024-04-19T01:46:40.247 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:40.247 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:40.247 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:40.248 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:40.248 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:40.248 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:40.248 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:40.248 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:40.248 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:40.248 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:40.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.243+0000 7f18f8de6640 10 monclient: _renew_subs 2024-04-19T01:46:40.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.243+0000 7f18f8de6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:46:40.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.243+0000 7f18f8de6640 1 -- 172.21.15.62:0/3946531235 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18f41d0740 con 0x7f18f40a2be0 2024-04-19T01:46:40.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.243+0000 7f18f8de6640 10 monclient: _renew_subs 2024-04-19T01:46:40.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.243+0000 7f18f8de6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:46:40.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.243+0000 7f18f8de6640 1 -- 172.21.15.62:0/3946531235 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18f411ea80 con 0x7f18f40a2be0 2024-04-19T01:46:40.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.243+0000 7f18f8de6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:40.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.243+0000 7f18f8de6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:40.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.243+0000 7f18f8de6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:46:40.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.243+0000 7f18f8de6640 1 -- 172.21.15.62:0/3946531235 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18f40692b0 con 0x7f18f40a2be0 2024-04-19T01:46:40.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.243+0000 7f18eaffd640 1 -- 172.21.15.62:0/3946531235 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f18dc013430 con 0x7f18f40a2be0 2024-04-19T01:46:40.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.243+0000 7f18eaffd640 1 --2- 172.21.15.62:0/3946531235 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f18d4036420 0x7f18d40388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:40.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.243+0000 7f18eaffd640 1 -- 172.21.15.62:0/3946531235 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(215..215 src has 108..215) ==== 4738+0+0 (secure 0 0 0) 0x7f18dc049630 con 0x7f18f40a2be0 2024-04-19T01:46:40.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.243+0000 7f18f1d9b640 1 --2- 172.21.15.62:0/3946531235 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f18d4036420 0x7f18d40388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:40.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.243+0000 7f18f1d9b640 10 monclient: get_auth_request con 0x7f18d4036420 auth_method 0 2024-04-19T01:46:40.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.247+0000 7f18f1d9b640 1 --2- 172.21.15.62:0/3946531235 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f18d4036420 0x7f18d40388e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f18e40012f0 tx=0x7f18e4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:40.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.247+0000 7f18eaffd640 1 -- 172.21.15.62:0/3946531235 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f18dc012550 con 0x7f18f40a2be0 2024-04-19T01:46:40.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.247+0000 7f18eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:40.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.247+0000 7f18eaffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:46:40.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.475+0000 7f18f8de6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:46:40.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.475+0000 7f18f8de6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:46:40.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.475+0000 7f18f8de6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:46:40.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.475+0000 7f18f8de6640 1 -- 172.21.15.62:0/3946531235 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f18f40761f0 con 0x7f18f40a2be0 2024-04-19T01:46:40.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.475+0000 7f18eaffd640 1 -- 172.21.15.62:0/3946531235 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f18dc0127b0 con 0x7f18f40a2be0 2024-04-19T01:46:40.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.475+0000 7f18eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:46:40.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.475+0000 7f18eaffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:46:40.473 INFO:teuthology.orchestra.run.smithi062.stdout:{"election_epoch":32,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":97,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","modified":"2024-04-19T01:41:38.497795Z","created":"2024-04-19T01:41:38.497795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:46:40.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.475+0000 7f18f8de6640 1 -- 172.21.15.62:0/3946531235 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f18d4036420 msgr2=0x7f18d40388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:40.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.475+0000 7f18f8de6640 1 --2- 172.21.15.62:0/3946531235 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f18d4036420 0x7f18d40388e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f18e40012f0 tx=0x7f18e4009040 comp rx=0 tx=0).stop 2024-04-19T01:46:40.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.475+0000 7f18f8de6640 10 monclient: shutdown 2024-04-19T01:46:40.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.475+0000 7f18f8de6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:46:40.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.475+0000 7f18f8de6640 1 -- 172.21.15.62:0/3946531235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f40a2be0 msgr2=0x7f18f41cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:40.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.475+0000 7f18f8de6640 1 --2- 172.21.15.62:0/3946531235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f40a2be0 0x7f18f41cf850 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f18dc00e810 tx=0x7f18dc00ecd0 comp rx=0 tx=0).stop 2024-04-19T01:46:40.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.475+0000 7f18f8de6640 1 -- 172.21.15.62:0/3946531235 shutdown_connections 2024-04-19T01:46:40.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.475+0000 7f18f8de6640 1 --2- 172.21.15.62:0/3946531235 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f18d4036420 0x7f18d40388e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:40.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.475+0000 7f18f8de6640 1 --2- 172.21.15.62:0/3946531235 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f18f4137cb0 0x7f18f411d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:40.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.475+0000 7f18f8de6640 1 --2- 172.21.15.62:0/3946531235 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18f41378d0 0x7f18f4136430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:40.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.475+0000 7f18f8de6640 1 --2- 172.21.15.62:0/3946531235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18f40a2be0 0x7f18f41cf850 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:40.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.475+0000 7f18f8de6640 1 -- 172.21.15.62:0/3946531235 >> 172.21.15.62:0/3946531235 conn(0x7f18f40a35c0 msgr2=0x7f18f4070160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:46:40.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.475+0000 7f18f8de6640 1 -- 172.21.15.62:0/3946531235 shutdown_connections 2024-04-19T01:46:40.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:40.475+0000 7f18f8de6640 1 -- 172.21.15.62:0/3946531235 wait complete. 2024-04-19T01:46:40.553 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:40.553 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30345]: starting. 2024-04-19T01:46:40.553 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30345]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-30309 2024-04-19T01:46:40.553 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30345]: shutting down. 2024-04-19T01:46:40.557 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:40.557 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:40.557 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:46:40.560 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:46:43.014 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30397]: starting. 2024-04-19T01:46:43.014 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30397]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-30396 2024-04-19T01:46:43.014 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30397]: created object 0... 2024-04-19T01:46:43.014 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30397]: created object 25... 2024-04-19T01:46:43.014 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30397]: created object 49... 2024-04-19T01:46:43.015 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30397]: finishing. 2024-04-19T01:46:43.015 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30397]: shutting down. 2024-04-19T01:46:43.493 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:46:43.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d1ce1a640 10 monclient: get_monmap_and_config 2024-04-19T01:46:43.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d1ce1a640 10 monclient: build_initial_monmap 2024-04-19T01:46:43.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d1ce1a640 10 monclient: monmap: 2024-04-19T01:46:43.626 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:46:43.626 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:43.626 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:46:43.631675+0000 2024-04-19T01:46:43.626 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:46:43.631675+0000 2024-04-19T01:46:43.626 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:46:43.626 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:43.626 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:46:43.626 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:46:43.626 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:46:43.626 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:46:43.627 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:46:43.627 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:43.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d1ce1a640 1 Processor -- start 2024-04-19T01:46:43.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d1ce1a640 1 -- start start 2024-04-19T01:46:43.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d1ce1a640 10 monclient: init 2024-04-19T01:46:43.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d1ce1a640 10 monclient: _reopen_session rank -1 2024-04-19T01:46:43.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d1ce1a640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-04-19T01:46:43.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d1ce1a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9d180a47e0 addr v1:172.21.15.71:6789/0 2024-04-19T01:46:43.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d1ce1a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d180a52d0 0x7f9d1813a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:43.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d1ce1a640 10 monclient(hunting): picked mon.noname-b con 0x7f9d180a52d0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:46:43.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d1ce1a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9d1813ab20 addr v1:172.21.15.62:6789/0 2024-04-19T01:46:43.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d1ce1a640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d1809f7c0 con 0x7f9d1813ab20 2024-04-19T01:46:43.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d1ce1a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d1809c2d0 con 0x7f9d180a47e0 2024-04-19T01:46:43.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d1ce1a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:43.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d1ce1a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f9d1809bf00 con 0x7f9d180a52d0 2024-04-19T01:46:43.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d1ce1a640 10 monclient(hunting): _renew_subs 2024-04-19T01:46:43.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d1ce1a640 10 monclient(hunting): authenticate will time out at 1189.451102s 2024-04-19T01:46:43.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d16d9d640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f9d1813ab20 0x7f9d1813cf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:60082/0 (socket says 172.21.15.62:60082) 2024-04-19T01:46:43.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d16d9d640 1 -- 172.21.15.62:0/3986995658 learned_addr learned my addr 172.21.15.62:0/3986995658 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:46:43.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d15d9b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d180a52d0 0x7f9d1813a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:43.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d15d9b640 10 monclient(hunting): get_auth_request con 0x7f9d180a52d0 auth_method 0 2024-04-19T01:46:43.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d15d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:43.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d15d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:43.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:43.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d15d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:43.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d15d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:43.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d15d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:43.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d15d9b640 10 monclient(hunting): handle_auth_done global_id 5249 payload 274 2024-04-19T01:46:43.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d15d9b640 1 -- 172.21.15.62:0/3986995658 >> v1:172.21.15.71:6789/0 conn(0x7f9d180a47e0 legacy=0x7f9d180a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:46:43.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d15d9b640 1 -- 172.21.15.62:0/3986995658 >> v1:172.21.15.62:6789/0 conn(0x7f9d1813ab20 legacy=0x7f9d1813cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:46:43.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d15d9b640 10 monclient: _finish_hunting 0 2024-04-19T01:46:43.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d15d9b640 1 monclient: found mon.noname-b 2024-04-19T01:46:43.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d15d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:46:43.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d15d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:46:43.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.627+0000 7f9d15d9b640 1 -- 172.21.15.62:0/3986995658 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d1809faa0 con 0x7f9d180a52d0 2024-04-19T01:46:43.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.631+0000 7f9d15d9b640 1 -- 172.21.15.62:0/3986995658 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d180a52d0 msgr2=0x7f9d1813a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:43.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.631+0000 7f9d15d9b640 1 --2- 172.21.15.62:0/3986995658 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d180a52d0 0x7f9d1813a5e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9d080097f0 tx=0x7f9d0801c250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7553fa77e3adc58 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:43.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.631+0000 7f9d1559a640 1 -- 172.21.15.62:0/3986995658 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d080b0070 con 0x7f9d180a52d0 2024-04-19T01:46:43.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:43.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:46:43.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1559a640 10 monclient: dump: 2024-04-19T01:46:43.632 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:43.632 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:43.633 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:43.633 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:43.633 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:43.633 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:43.633 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:43.633 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:43.633 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:43.633 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:43.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1559a640 10 monclient: _finish_auth 0 2024-04-19T01:46:43.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1559a640 10 monclient: _check_auth_tickets 2024-04-19T01:46:43.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:46:43.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1559a640 1 -- 172.21.15.62:0/3986995658 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d080ab600 con 0x7f9d180a52d0 2024-04-19T01:46:43.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 5 monclient: authenticate success, global_id 5249 2024-04-19T01:46:43.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1559a640 10 monclient: handle_config config(0 keys) 2024-04-19T01:46:43.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1559a640 1 -- 172.21.15.62:0/3986995658 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d080ab9d0 con 0x7f9d180a52d0 2024-04-19T01:46:43.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 10 monclient: get_monmap_and_config success 2024-04-19T01:46:43.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 4 set_mon_vals no callback set 2024-04-19T01:46:43.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:43.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:46:43.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1559a640 10 monclient: dump: 2024-04-19T01:46:43.634 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:43.634 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:43.634 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:43.634 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:43.634 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:43.634 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:43.634 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:43.634 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:43.635 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:43.635 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:43.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 10 monclient: shutdown 2024-04-19T01:46:43.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:46:43.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 1 -- 172.21.15.62:0/3986995658 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d180a52d0 msgr2=0x7f9d1813a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:43.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 1 --2- 172.21.15.62:0/3986995658 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d180a52d0 0x7f9d1813a5e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9d080097f0 tx=0x7f9d0801c250 comp rx=0 tx=0).stop 2024-04-19T01:46:43.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 1 -- 172.21.15.62:0/3986995658 shutdown_connections 2024-04-19T01:46:43.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 1 --2- 172.21.15.62:0/3986995658 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d180a52d0 0x7f9d1813a5e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:43.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 1 -- 172.21.15.62:0/3986995658 >> 172.21.15.62:0/3986995658 conn(0x7f9d180a23d0 msgr2=0x7f9d180a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:46:43.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 1 -- 172.21.15.62:0/3986995658 shutdown_connections 2024-04-19T01:46:43.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 1 -- 172.21.15.62:0/3986995658 wait complete. 2024-04-19T01:46:43.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 10 monclient: build_initial_monmap 2024-04-19T01:46:43.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 10 monclient: monmap: 2024-04-19T01:46:43.639 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:46:43.639 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:46:43.639 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:46:43.639 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:46:43.639 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:46:43.639 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:43.639 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:46:43.639 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:46:43.639 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:46:43.639 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:43.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 1 Processor -- start 2024-04-19T01:46:43.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 1 -- start start 2024-04-19T01:46:43.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 10 monclient: init 2024-04-19T01:46:43.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 10 monclient: _reopen_session rank -1 2024-04-19T01:46:43.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-19T01:46:43.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d180a47e0 0x7f9d181cd380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:43.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 10 monclient(hunting): picked mon.noname-b con 0x7f9d180a47e0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:46:43.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d180a52d0 0x7f9d181cd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:43.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 10 monclient(hunting): picked mon.noname-c con 0x7f9d180a52d0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:46:43.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d1813ab20 0x7f9d181cde00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:43.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 10 monclient(hunting): picked mon.noname-a con 0x7f9d1813ab20 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:46:43.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:43.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9d18140e50 con 0x7f9d1813ab20 2024-04-19T01:46:43.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:43.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1809c970 con 0x7f9d180a47e0 2024-04-19T01:46:43.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:43.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f9d18140fd0 con 0x7f9d180a52d0 2024-04-19T01:46:43.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 10 monclient(hunting): _renew_subs 2024-04-19T01:46:43.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1ce1a640 10 monclient(hunting): authenticate will time out at 1189.458261s 2024-04-19T01:46:43.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d16d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d1813ab20 0x7f9d181cde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:43.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d15d9b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d180a52d0 0x7f9d181cd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:43.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d15d9b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d180a52d0 0x7f9d181cd8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:50704/0 (socket says 172.21.15.62:50704) 2024-04-19T01:46:43.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d15d9b640 1 -- 172.21.15.62:0/1473226710 learned_addr learned my addr 172.21.15.62:0/1473226710 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:46:43.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d15d9b640 10 monclient(hunting): get_auth_request con 0x7f9d180a52d0 auth_method 0 2024-04-19T01:46:43.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d15d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:43.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d15d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:43.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:43.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d16d9d640 1 --2- 172.21.15.62:0/1473226710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d1813ab20 0x7f9d181cde00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57168/0 (socket says 172.21.15.62:57168) 2024-04-19T01:46:43.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d16d9d640 10 monclient(hunting): get_auth_request con 0x7f9d1813ab20 auth_method 0 2024-04-19T01:46:43.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d16d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:43.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d16d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:43.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d16d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:43.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d15d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:43.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d15d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:43.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1659c640 1 --2- 172.21.15.62:0/1473226710 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d180a47e0 0x7f9d181cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:43.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d15d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:43.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d16d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:43.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d16d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:43.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d16d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:43.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1659c640 10 monclient(hunting): get_auth_request con 0x7f9d180a47e0 auth_method 0 2024-04-19T01:46:43.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:43.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:43.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:43.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d15d9b640 10 monclient(hunting): handle_auth_done global_id 5252 payload 824 2024-04-19T01:46:43.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1659c640 1 -- 172.21.15.62:0/1473226710 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d180a47e0 msgr2=0x7f9d181cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-19T01:46:43.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1659c640 1 -- 172.21.15.62:0/1473226710 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d180a47e0 msgr2=0x7f9d181cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:46:43.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d15d9b640 1 -- 172.21.15.62:0/1473226710 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d180a47e0 msgr2=0x7f9d181cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:43.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1659c640 1 --2- 172.21.15.62:0/1473226710 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d180a47e0 0x7f9d181cd380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:46:43.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d1659c640 1 --2- 172.21.15.62:0/1473226710 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d180a47e0 0x7f9d181cd380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:46:43.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d15d9b640 1 --2- 172.21.15.62:0/1473226710 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d180a47e0 0x7f9d181cd380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:43.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d15d9b640 1 -- 172.21.15.62:0/1473226710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d1813ab20 msgr2=0x7f9d181cde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:43.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d15d9b640 1 --2- 172.21.15.62:0/1473226710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d1813ab20 0x7f9d181cde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:43.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d15d9b640 10 monclient: _finish_hunting 0 2024-04-19T01:46:43.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d15d9b640 1 monclient: found mon.noname-c 2024-04-19T01:46:43.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d15d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:46:43.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d15d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:46:43.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d15d9b640 1 -- 172.21.15.62:0/1473226710 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d1811eb20 con 0x7f9d180a52d0 2024-04-19T01:46:43.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.635+0000 7f9d16d9d640 1 --2- 172.21.15.62:0/1473226710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d1813ab20 0x7f9d181cde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:46:43.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.639+0000 7f9d15d9b640 1 -- 172.21.15.62:0/1473226710 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d180a52d0 msgr2=0x7f9d181cd8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:43.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.639+0000 7f9d15d9b640 1 --2- 172.21.15.62:0/1473226710 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d180a52d0 0x7f9d181cd8c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9d080ae040 tx=0x7f9d0801aa90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:43.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.663+0000 7f9d06ffd640 1 -- 172.21.15.62:0/1473226710 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d080b0050 con 0x7f9d180a52d0 2024-04-19T01:46:43.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.663+0000 7f9d06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:43.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.663+0000 7f9d06ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:46:43.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.663+0000 7f9d06ffd640 10 monclient: dump: 2024-04-19T01:46:43.661 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:43.661 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:43.661 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:43.661 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:43.661 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:43.661 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:43.662 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:43.662 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:43.662 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:43.662 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:43.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.663+0000 7f9d06ffd640 10 monclient: _finish_auth 0 2024-04-19T01:46:43.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.663+0000 7f9d06ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:46:43.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.663+0000 7f9d06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:46:43.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.663+0000 7f9d06ffd640 1 -- 172.21.15.62:0/1473226710 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d080bc070 con 0x7f9d180a52d0 2024-04-19T01:46:43.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.663+0000 7f9d06ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:46:43.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.663+0000 7f9d06ffd640 1 -- 172.21.15.62:0/1473226710 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d080b5c80 con 0x7f9d180a52d0 2024-04-19T01:46:43.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.663+0000 7f9d06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:43.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.663+0000 7f9d06ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:46:43.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.663+0000 7f9d06ffd640 10 monclient: dump: 2024-04-19T01:46:43.663 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:43.663 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:43.663 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:43.663 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:43.663 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:43.663 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:43.663 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:43.663 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:43.663 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:43.663 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:43.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.663+0000 7f9d07fff640 4 set_mon_vals no callback set 2024-04-19T01:46:43.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.663+0000 7f9d1ce1a640 5 monclient: authenticate success, global_id 5252 2024-04-19T01:46:43.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.663+0000 7f9d1ce1a640 10 monclient: _renew_subs 2024-04-19T01:46:43.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.663+0000 7f9d1ce1a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:46:43.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.663+0000 7f9d1ce1a640 1 -- 172.21.15.62:0/1473226710 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d1811e280 con 0x7f9d180a52d0 2024-04-19T01:46:43.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.663+0000 7f9d1ce1a640 10 monclient: _renew_subs 2024-04-19T01:46:43.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.663+0000 7f9d1ce1a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:46:43.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.663+0000 7f9d1ce1a640 1 -- 172.21.15.62:0/1473226710 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d1811d390 con 0x7f9d180a52d0 2024-04-19T01:46:43.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.663+0000 7f9d1ce1a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:43.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.663+0000 7f9d1ce1a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:43.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.663+0000 7f9d1ce1a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:46:43.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.663+0000 7f9d1ce1a640 1 -- 172.21.15.62:0/1473226710 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d180692b0 con 0x7f9d180a52d0 2024-04-19T01:46:43.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.663+0000 7f9d06ffd640 1 -- 172.21.15.62:0/1473226710 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f9d080ab4c0 con 0x7f9d180a52d0 2024-04-19T01:46:43.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.663+0000 7f9d06ffd640 1 --2- 172.21.15.62:0/1473226710 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f9cf803a880 0x7f9cf803cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:43.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.663+0000 7f9d06ffd640 1 -- 172.21.15.62:0/1473226710 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(218..218 src has 108..218) ==== 4738+0+0 (secure 0 0 0) 0x7f9d080e3d20 con 0x7f9d180a52d0 2024-04-19T01:46:43.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.663+0000 7f9d1659c640 1 --2- 172.21.15.62:0/1473226710 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f9cf803a880 0x7f9cf803cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:43.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.663+0000 7f9d1659c640 10 monclient: get_auth_request con 0x7f9cf803a880 auth_method 0 2024-04-19T01:46:43.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.667+0000 7f9d1659c640 1 --2- 172.21.15.62:0/1473226710 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f9cf803a880 0x7f9cf803cd40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c007fd0 tx=0x7f9d0c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:43.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.671+0000 7f9d06ffd640 1 -- 172.21.15.62:0/1473226710 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9d1809f7c0 con 0x7f9d180a52d0 2024-04-19T01:46:43.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.671+0000 7f9d06ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:43.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.671+0000 7f9d06ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:46:43.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.899+0000 7f9d1ce1a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:46:43.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.899+0000 7f9d1ce1a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:46:43.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.899+0000 7f9d1ce1a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:46:43.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.899+0000 7f9d1ce1a640 1 -- 172.21.15.62:0/1473226710 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9d18071890 con 0x7f9d180a52d0 2024-04-19T01:46:43.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.899+0000 7f9d06ffd640 1 -- 172.21.15.62:0/1473226710 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f9d080e2740 con 0x7f9d180a52d0 2024-04-19T01:46:43.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.899+0000 7f9d06ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:46:43.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.899+0000 7f9d06ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:46:43.899 INFO:teuthology.orchestra.run.smithi062.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","modified":"2024-04-19T01:41:38.497795Z","created":"2024-04-19T01:41:38.497795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:46:43.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.903+0000 7f9d1ce1a640 1 -- 172.21.15.62:0/1473226710 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f9cf803a880 msgr2=0x7f9cf803cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:43.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.903+0000 7f9d1ce1a640 1 --2- 172.21.15.62:0/1473226710 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f9cf803a880 0x7f9cf803cd40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c007fd0 tx=0x7f9d0c009040 comp rx=0 tx=0).stop 2024-04-19T01:46:43.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.903+0000 7f9d1ce1a640 10 monclient: shutdown 2024-04-19T01:46:43.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.903+0000 7f9d1ce1a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:46:43.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.903+0000 7f9d1ce1a640 1 -- 172.21.15.62:0/1473226710 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d180a52d0 msgr2=0x7f9d181cd8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:43.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.903+0000 7f9d1ce1a640 1 --2- 172.21.15.62:0/1473226710 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d180a52d0 0x7f9d181cd8c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9d080ae040 tx=0x7f9d0801aa90 comp rx=0 tx=0).stop 2024-04-19T01:46:43.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.903+0000 7f9d1ce1a640 1 -- 172.21.15.62:0/1473226710 shutdown_connections 2024-04-19T01:46:43.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.903+0000 7f9d1ce1a640 1 --2- 172.21.15.62:0/1473226710 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f9cf803a880 0x7f9cf803cd40 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:43.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.903+0000 7f9d1ce1a640 1 --2- 172.21.15.62:0/1473226710 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9d180a52d0 0x7f9d181cd8c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:43.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.903+0000 7f9d1ce1a640 1 --2- 172.21.15.62:0/1473226710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9d1813ab20 0x7f9d181cde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:43.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.903+0000 7f9d1ce1a640 1 --2- 172.21.15.62:0/1473226710 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d180a47e0 0x7f9d181cd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:43.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.903+0000 7f9d1ce1a640 1 -- 172.21.15.62:0/1473226710 >> 172.21.15.62:0/1473226710 conn(0x7f9d180a23d0 msgr2=0x7f9d1813cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:46:43.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.903+0000 7f9d1ce1a640 1 -- 172.21.15.62:0/1473226710 shutdown_connections 2024-04-19T01:46:43.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:43.903+0000 7f9d1ce1a640 1 -- 172.21.15.62:0/1473226710 wait complete. 2024-04-19T01:46:43.917 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-19T01:46:43.917 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-19T01:46:43.918 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-19T01:46:44.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa48b4e7640 10 monclient: get_monmap_and_config 2024-04-19T01:46:44.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa48b4e7640 10 monclient: build_initial_monmap 2024-04-19T01:46:44.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa48b4e7640 10 monclient: monmap: 2024-04-19T01:46:44.034 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:46:44.035 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:44.035 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:46:44.040167+0000 2024-04-19T01:46:44.035 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:46:44.040167+0000 2024-04-19T01:46:44.035 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:46:44.035 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:44.035 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:46:44.035 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:46:44.035 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:46:44.035 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:46:44.035 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:46:44.035 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:44.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa48b4e7640 1 Processor -- start 2024-04-19T01:46:44.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa48b4e7640 1 -- start start 2024-04-19T01:46:44.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa48b4e7640 10 monclient: init 2024-04-19T01:46:44.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa48b4e7640 10 monclient: _reopen_session rank -1 2024-04-19T01:46:44.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa48b4e7640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-04-19T01:46:44.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa48b4e7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4840a47e0 0x7fa48409f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:44.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa48b4e7640 10 monclient(hunting): picked mon.noname-b con 0x7fa4840a47e0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:46:44.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa48b4e7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa48409f890 addr v1:172.21.15.71:6789/0 2024-04-19T01:46:44.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa48b4e7640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa4840a52d0 0x7fa48413b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:44.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa48b4e7640 10 monclient(hunting): picked mon.noname-a con 0x7fa4840a52d0 addr v2:172.21.15.62:3300/0 2024-04-19T01:46:44.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa48b4e7640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa48409ec50 con 0x7fa48409f890 2024-04-19T01:46:44.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa48b4e7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:44.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa48b4e7640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fa48409ef30 con 0x7fa4840a52d0 2024-04-19T01:46:44.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa48b4e7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:44.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa48b4e7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fa48409e690 con 0x7fa4840a47e0 2024-04-19T01:46:44.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa48b4e7640 10 monclient(hunting): _renew_subs 2024-04-19T01:46:44.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa48b4e7640 10 monclient(hunting): authenticate will time out at 1189.859670s 2024-04-19T01:46:44.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa489283640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4840a47e0 0x7fa48409f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:44.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa489a84640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa4840a52d0 0x7fa48413b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:44.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa489283640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4840a47e0 0x7fa48409f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:50706/0 (socket says 172.21.15.62:50706) 2024-04-19T01:46:44.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa489283640 1 -- 172.21.15.62:0/1650507876 learned_addr learned my addr 172.21.15.62:0/1650507876 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:46:44.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa489283640 10 monclient(hunting): get_auth_request con 0x7fa4840a47e0 auth_method 0 2024-04-19T01:46:44.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa489283640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:44.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa489283640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:44.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa489283640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:44.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa489a84640 10 monclient(hunting): get_auth_request con 0x7fa4840a52d0 auth_method 0 2024-04-19T01:46:44.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa489a84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:44.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa489a84640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:44.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa489a84640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:44.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa489a84640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:44.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.035+0000 7fa489a84640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:44.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.039+0000 7fa489a84640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:44.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.039+0000 7fa489283640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:44.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.039+0000 7fa489283640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:44.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.039+0000 7fa489283640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:44.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.039+0000 7fa489a84640 10 monclient(hunting): handle_auth_done global_id 14715 payload 274 2024-04-19T01:46:44.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.039+0000 7fa489a84640 1 -- 172.21.15.62:0/1650507876 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4840a47e0 msgr2=0x7fa48409f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:46:44.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.039+0000 7fa489a84640 1 --2- 172.21.15.62:0/1650507876 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4840a47e0 0x7fa48409f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:44.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.039+0000 7fa489a84640 1 -- 172.21.15.62:0/1650507876 >> v1:172.21.15.71:6789/0 conn(0x7fa48409f890 legacy=0x7fa4840a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:46:44.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.039+0000 7fa489a84640 10 monclient: _finish_hunting 0 2024-04-19T01:46:44.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.039+0000 7fa489a84640 1 monclient: found mon.noname-a 2024-04-19T01:46:44.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.039+0000 7fa489a84640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:46:44.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.039+0000 7fa489a84640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:46:44.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.039+0000 7fa489a84640 1 -- 172.21.15.62:0/1650507876 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa48409e970 con 0x7fa4840a52d0 2024-04-19T01:46:44.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa489a84640 1 -- 172.21.15.62:0/1650507876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4840a52d0 msgr2=0x7fa48413b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:44.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa489a84640 1 --2- 172.21.15.62:0/1650507876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4840a52d0 0x7fa48413b570 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa47c009d90 tx=0x7fa47c01c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ba9a77c6fbfcf93 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:44.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa47bfff640 1 -- 172.21.15.62:0/1650507876 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa47c0b2070 con 0x7fa4840a52d0 2024-04-19T01:46:44.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa47bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:44.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa47bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:46:44.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa47bfff640 10 monclient: dump: 2024-04-19T01:46:44.049 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:44.049 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:44.049 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:44.049 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:44.049 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:44.049 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:44.049 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:44.049 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:44.049 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:44.049 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:44.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa47bfff640 10 monclient: _finish_auth 0 2024-04-19T01:46:44.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa47bfff640 10 monclient: _check_auth_tickets 2024-04-19T01:46:44.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa47bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:46:44.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa47bfff640 1 -- 172.21.15.62:0/1650507876 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa47c0a5920 con 0x7fa4840a52d0 2024-04-19T01:46:44.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 5 monclient: authenticate success, global_id 14715 2024-04-19T01:46:44.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:46:44.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa47bfff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:46:44.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa47bfff640 1 -- 172.21.15.62:0/1650507876 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa47c0a5cf0 con 0x7fa4840a52d0 2024-04-19T01:46:44.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 10 monclient: get_monmap_and_config success 2024-04-19T01:46:44.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 4 set_mon_vals no callback set 2024-04-19T01:46:44.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa47bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:44.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa47bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:46:44.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa47bfff640 10 monclient: dump: 2024-04-19T01:46:44.050 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:44.050 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:44.050 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:44.050 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:44.050 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:44.051 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:44.051 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:44.051 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:44.051 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:44.051 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:44.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 10 monclient: shutdown 2024-04-19T01:46:44.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:46:44.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 1 -- 172.21.15.62:0/1650507876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4840a52d0 msgr2=0x7fa48413b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:44.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 1 --2- 172.21.15.62:0/1650507876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4840a52d0 0x7fa48413b570 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa47c009d90 tx=0x7fa47c01c8c0 comp rx=0 tx=0).stop 2024-04-19T01:46:44.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 1 -- 172.21.15.62:0/1650507876 shutdown_connections 2024-04-19T01:46:44.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 1 --2- 172.21.15.62:0/1650507876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4840a52d0 0x7fa48413b570 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:44.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 1 --2- 172.21.15.62:0/1650507876 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4840a47e0 0x7fa48409f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:44.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 1 -- 172.21.15.62:0/1650507876 >> 172.21.15.62:0/1650507876 conn(0x7fa4840a23d0 msgr2=0x7fa4840a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:46:44.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 1 -- 172.21.15.62:0/1650507876 shutdown_connections 2024-04-19T01:46:44.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 1 -- 172.21.15.62:0/1650507876 wait complete. 2024-04-19T01:46:44.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 10 monclient: build_initial_monmap 2024-04-19T01:46:44.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 10 monclient: monmap: 2024-04-19T01:46:44.052 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:46:44.052 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:46:44.052 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:46:44.052 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:46:44.052 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:46:44.052 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:44.052 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:46:44.052 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:46:44.052 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:46:44.052 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:44.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 1 Processor -- start 2024-04-19T01:46:44.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 1 -- start start 2024-04-19T01:46:44.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 10 monclient: init 2024-04-19T01:46:44.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 10 monclient: _reopen_session rank -1 2024-04-19T01:46:44.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-19T01:46:44.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa48409f890 0x7fa484123ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:44.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 10 monclient(hunting): picked mon.noname-a con 0x7fa48409f890 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:46:44.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4840a47e0 0x7fa4841cf810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:44.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 10 monclient(hunting): picked mon.noname-b con 0x7fa4840a47e0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:46:44.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4840a52d0 0x7fa484137880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:44.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 10 monclient(hunting): picked mon.noname-c con 0x7fa4840a52d0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:46:44.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:44.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa48413e800 con 0x7fa48409f890 2024-04-19T01:46:44.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:44.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa48413e680 con 0x7fa4840a47e0 2024-04-19T01:46:44.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:44.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fa48413e980 con 0x7fa4840a52d0 2024-04-19T01:46:44.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 10 monclient(hunting): _renew_subs 2024-04-19T01:46:44.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa48b4e7640 10 monclient(hunting): authenticate will time out at 1189.866851s 2024-04-19T01:46:44.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa489a84640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4840a52d0 0x7fa484137880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:44.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa489283640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa48409f890 0x7fa484123ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:44.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa489a84640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4840a52d0 0x7fa484137880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:50718/0 (socket says 172.21.15.62:50718) 2024-04-19T01:46:44.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa489a84640 1 -- 172.21.15.62:0/4025171904 learned_addr learned my addr 172.21.15.62:0/4025171904 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:46:44.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa489283640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa48409f890 0x7fa484123ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57192/0 (socket says 172.21.15.62:57192) 2024-04-19T01:46:44.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa489a84640 10 monclient(hunting): get_auth_request con 0x7fa4840a52d0 auth_method 0 2024-04-19T01:46:44.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa489a84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:44.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa489a84640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:44.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa489a84640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:44.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa489283640 10 monclient(hunting): get_auth_request con 0x7fa48409f890 auth_method 0 2024-04-19T01:46:44.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa489283640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:44.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa489283640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:44.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa489283640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:44.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa489a84640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:44.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa489a84640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:44.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa489a84640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:44.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa489283640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:44.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa489283640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:44.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa489283640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:44.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa488a82640 1 --2- 172.21.15.62:0/4025171904 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4840a47e0 0x7fa4841cf810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:44.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa489a84640 10 monclient(hunting): handle_auth_done global_id 5258 payload 824 2024-04-19T01:46:44.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa489a84640 1 -- 172.21.15.62:0/4025171904 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4840a47e0 msgr2=0x7fa4841cf810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:44.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa489a84640 1 --2- 172.21.15.62:0/4025171904 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4840a47e0 0x7fa4841cf810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:44.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa489a84640 1 -- 172.21.15.62:0/4025171904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa48409f890 msgr2=0x7fa484123ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:44.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa489a84640 1 --2- 172.21.15.62:0/4025171904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa48409f890 0x7fa484123ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:44.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa489a84640 10 monclient: _finish_hunting 0 2024-04-19T01:46:44.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa489a84640 1 monclient: found mon.noname-c 2024-04-19T01:46:44.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa489a84640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:46:44.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa489a84640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:46:44.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa489a84640 1 -- 172.21.15.62:0/4025171904 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa48409e000 con 0x7fa4840a52d0 2024-04-19T01:46:44.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa488a82640 10 monclient: get_auth_request con 0x7fa4840a47e0 auth_method 0 2024-04-19T01:46:44.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa489283640 1 --2- 172.21.15.62:0/4025171904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa48409f890 0x7fa484123ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:46:44.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.043+0000 7fa488a82640 1 --2- 172.21.15.62:0/4025171904 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4840a47e0 0x7fa4841cf810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:46:44.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa489a84640 1 -- 172.21.15.62:0/4025171904 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4840a52d0 msgr2=0x7fa484137880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:44.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa489a84640 1 --2- 172.21.15.62:0/4025171904 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4840a52d0 0x7fa484137880 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa47c01c6f0 tx=0x7fa47c019660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:44.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa479ffb640 1 -- 172.21.15.62:0/4025171904 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa47c0bd020 con 0x7fa4840a52d0 2024-04-19T01:46:44.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa479ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:44.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa479ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:46:44.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa479ffb640 10 monclient: dump: 2024-04-19T01:46:44.056 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:44.056 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:44.056 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:44.056 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:44.056 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:44.056 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:44.056 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:44.056 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:44.057 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:44.057 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:44.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa479ffb640 10 monclient: _finish_auth 0 2024-04-19T01:46:44.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa479ffb640 10 monclient: _check_auth_tickets 2024-04-19T01:46:44.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa479ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:46:44.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa479ffb640 1 -- 172.21.15.62:0/4025171904 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa47c0aeb90 con 0x7fa4840a52d0 2024-04-19T01:46:44.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa479ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:46:44.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa479ffb640 1 -- 172.21.15.62:0/4025171904 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa47c0b2040 con 0x7fa4840a52d0 2024-04-19T01:46:44.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa479ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:44.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa479ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:46:44.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa479ffb640 10 monclient: dump: 2024-04-19T01:46:44.057 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:44.057 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:44.057 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:44.057 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:44.057 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:44.058 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:44.058 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:44.058 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:44.058 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:44.058 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:44.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa47b7fe640 4 set_mon_vals no callback set 2024-04-19T01:46:44.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa48b4e7640 5 monclient: authenticate success, global_id 5258 2024-04-19T01:46:44.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa48b4e7640 10 monclient: _renew_subs 2024-04-19T01:46:44.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa48b4e7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:46:44.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa48b4e7640 1 -- 172.21.15.62:0/4025171904 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa48406f5f0 con 0x7fa4840a52d0 2024-04-19T01:46:44.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa48b4e7640 10 monclient: _renew_subs 2024-04-19T01:46:44.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa48b4e7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:46:44.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa48b4e7640 1 -- 172.21.15.62:0/4025171904 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa484137dc0 con 0x7fa4840a52d0 2024-04-19T01:46:44.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa48b4e7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:44.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa48b4e7640 1 --2- 172.21.15.62:0/4025171904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa484124a30 0x7fa4841cc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:44.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa48b4e7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:44.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa48b4e7640 1 -- 172.21.15.62:0/4025171904 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa484138200 con 0x7fa484124a30 2024-04-19T01:46:44.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa479ffb640 1 -- 172.21.15.62:0/4025171904 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fa47c0c0020 con 0x7fa4840a52d0 2024-04-19T01:46:44.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa489283640 1 --2- 172.21.15.62:0/4025171904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa484124a30 0x7fa4841cc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:44.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa489283640 10 monclient: get_auth_request con 0x7fa484124a30 auth_method 0 2024-04-19T01:46:44.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa489283640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:44.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa489283640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:44.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa489283640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:44.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa489283640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:44.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa489283640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:44.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa489283640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:44.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa489283640 10 monclient(hunting): handle_auth_done global_id 14721 payload 824 2024-04-19T01:46:44.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa479ffb640 1 --2- 172.21.15.62:0/4025171904 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa4680364a0 0x7fa468038960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:44.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa489283640 1 -- 172.21.15.62:0/4025171904 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa4840692b0 con 0x7fa484124a30 2024-04-19T01:46:44.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa479ffb640 1 -- 172.21.15.62:0/4025171904 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(218..218 src has 108..218) ==== 4738+0+0 (secure 0 0 0) 0x7fa47c0e4810 con 0x7fa4840a52d0 2024-04-19T01:46:44.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.047+0000 7fa488a82640 1 --2- 172.21.15.62:0/4025171904 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa4680364a0 0x7fa468038960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:44.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.051+0000 7fa488a82640 10 monclient: get_auth_request con 0x7fa4680364a0 auth_method 0 2024-04-19T01:46:44.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.051+0000 7fa489283640 1 -- 172.21.15.62:0/4025171904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa484124a30 msgr2=0x7fa4841cc5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:44.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.051+0000 7fa489283640 1 --2- 172.21.15.62:0/4025171904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa484124a30 0x7fa4841cc5d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa480006cd0 tx=0x7fa48000f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:44.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.051+0000 7fa479ffb640 1 -- 172.21.15.62:0/4025171904 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa480004020 con 0x7fa484124a30 2024-04-19T01:46:44.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.051+0000 7fa479ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:44.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.051+0000 7fa479ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:46:44.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.051+0000 7fa479ffb640 10 monclient: dump: 2024-04-19T01:46:44.062 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:44.062 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:44.062 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:44.062 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:44.063 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:44.063 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:44.063 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:44.063 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:44.063 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:44.063 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:44.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.051+0000 7fa479ffb640 1 -- 172.21.15.62:0/4025171904 <== mon.0 v2:172.21.15.62:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa480004020 con 0x7fa484124a30 2024-04-19T01:46:44.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.051+0000 7fa479ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:44.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.051+0000 7fa479ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:46:44.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.051+0000 7fa479ffb640 1 -- 172.21.15.62:0/4025171904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa484124a30 msgr2=0x7fa4841cc5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:44.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.051+0000 7fa479ffb640 1 --2- 172.21.15.62:0/4025171904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa484124a30 0x7fa4841cc5d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa480006cd0 tx=0x7fa48000f610 comp rx=0 tx=0).stop 2024-04-19T01:46:44.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.051+0000 7fa479ffb640 1 -- 172.21.15.62:0/4025171904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa484124a30 msgr2=0x7fa4841cc5d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:46:44.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.051+0000 7fa479ffb640 1 --2- 172.21.15.62:0/4025171904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa484124a30 0x7fa4841cc5d0 secure :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa480006cd0 tx=0x7fa48000f610 comp rx=0 tx=0).stop 2024-04-19T01:46:44.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.051+0000 7fa488a82640 1 --2- 172.21.15.62:0/4025171904 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa4680364a0 0x7fa468038960 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa4841235c0 tx=0x7fa474009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:44.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.059+0000 7fa48b4e7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:46:44.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.059+0000 7fa48b4e7640 1 --2- 172.21.15.62:0/4025171904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa48402c450 0x7fa48402c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:44.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.059+0000 7fa48b4e7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:44.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.059+0000 7fa48b4e7640 1 -- 172.21.15.62:0/4025171904 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa48406ee30 con 0x7fa48402c450 2024-04-19T01:46:44.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.059+0000 7fa489283640 1 --2- 172.21.15.62:0/4025171904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa48402c450 0x7fa48402c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:44.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.059+0000 7fa489283640 10 monclient: get_auth_request con 0x7fa48402c450 auth_method 0 2024-04-19T01:46:44.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.059+0000 7fa489283640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:44.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.059+0000 7fa489283640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:44.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.059+0000 7fa489283640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:44.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.059+0000 7fa489283640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:44.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.059+0000 7fa489283640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:44.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.059+0000 7fa489283640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:44.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.059+0000 7fa489283640 10 monclient(hunting): handle_auth_done global_id 14724 payload 824 2024-04-19T01:46:44.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.059+0000 7fa489283640 1 -- 172.21.15.62:0/4025171904 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa48413b240 con 0x7fa48402c450 2024-04-19T01:46:44.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.059+0000 7fa489283640 1 -- 172.21.15.62:0/4025171904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa48402c450 msgr2=0x7fa48402c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:44.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.059+0000 7fa489283640 1 --2- 172.21.15.62:0/4025171904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa48402c450 0x7fa48402c860 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa480009fd0 tx=0x7fa4800046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:44.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.059+0000 7fa479ffb640 1 -- 172.21.15.62:0/4025171904 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa480003a50 con 0x7fa48402c450 2024-04-19T01:46:44.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.059+0000 7fa479ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:44.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.059+0000 7fa479ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:46:44.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.059+0000 7fa479ffb640 10 monclient: dump: 2024-04-19T01:46:44.065 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:44.065 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:44.065 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:44.065 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:44.065 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:44.065 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:44.065 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:44.065 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:44.065 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:44.065 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:44.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.059+0000 7fa479ffb640 1 -- 172.21.15.62:0/4025171904 <== mon.0 v2:172.21.15.62:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4719 (secure 0 0 0) 0x7fa480003a30 con 0x7fa48402c450 2024-04-19T01:46:44.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.059+0000 7fa479ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:46:44.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.059+0000 7fa479ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:46:44.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.059+0000 7fa479ffb640 1 -- 172.21.15.62:0/4025171904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa48402c450 msgr2=0x7fa48402c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:44.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.059+0000 7fa479ffb640 1 --2- 172.21.15.62:0/4025171904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa48402c450 0x7fa48402c860 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa480009fd0 tx=0x7fa4800046d0 comp rx=0 tx=0).stop 2024-04-19T01:46:44.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.059+0000 7fa479ffb640 1 -- 172.21.15.62:0/4025171904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa48402c450 msgr2=0x7fa48402c860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:46:44.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.059+0000 7fa479ffb640 1 --2- 172.21.15.62:0/4025171904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa48402c450 0x7fa48402c860 secure :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa480009fd0 tx=0x7fa4800046d0 comp rx=0 tx=0).stop 2024-04-19T01:46:44.066 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-19T01:46:44.066 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "a", 2024-04-19T01:46:44.066 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 0, 2024-04-19T01:46:44.066 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "leader", 2024-04-19T01:46:44.066 INFO:teuthology.orchestra.run.smithi062.stdout: "election_epoch": 34, 2024-04-19T01:46:44.067 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 147622, 2024-04-19T01:46:44.067 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum": [ 2024-04-19T01:46:44.067 INFO:teuthology.orchestra.run.smithi062.stdout: 0, 2024-04-19T01:46:44.067 INFO:teuthology.orchestra.run.smithi062.stdout: 1, 2024-04-19T01:46:44.067 INFO:teuthology.orchestra.run.smithi062.stdout: 2 2024-04-19T01:46:44.067 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:46:44.067 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_age": 0, 2024-04-19T01:46:44.067 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:46:44.067 INFO:teuthology.orchestra.run.smithi062.stdout: "required_con": "2450521708007866372", 2024-04-19T01:46:44.067 INFO:teuthology.orchestra.run.smithi062.stdout: "required_mon": [ 2024-04-19T01:46:44.067 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:46:44.067 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:46:44.067 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:46:44.067 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:46:44.067 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:46:44.067 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:46:44.068 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:46:44.068 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:46:44.068 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:46:44.068 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:46:44.068 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:46:44.068 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:46:44.068 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:46:44.068 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_mon": [ 2024-04-19T01:46:44.068 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:46:44.068 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:46:44.068 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:46:44.068 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:46:44.068 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:46:44.068 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:46:44.069 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:46:44.069 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:46:44.069 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:46:44.069 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:46:44.069 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:46:44.069 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:44.069 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:44.069 INFO:teuthology.orchestra.run.smithi062.stdout: "outside_quorum": [], 2024-04-19T01:46:44.069 INFO:teuthology.orchestra.run.smithi062.stdout: "extra_probe_peers": [], 2024-04-19T01:46:44.069 INFO:teuthology.orchestra.run.smithi062.stdout: "sync_provider": [], 2024-04-19T01:46:44.069 INFO:teuthology.orchestra.run.smithi062.stdout: "monmap": { 2024-04-19T01:46:44.069 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 1, 2024-04-19T01:46:44.069 INFO:teuthology.orchestra.run.smithi062.stdout: "fsid": "6466e4b6-2520-485f-95b3-9401915fbbad", 2024-04-19T01:46:44.069 INFO:teuthology.orchestra.run.smithi062.stdout: "modified": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:46:44.069 INFO:teuthology.orchestra.run.smithi062.stdout: "created": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:46:44.070 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release": 19, 2024-04-19T01:46:44.070 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release_name": "squid", 2024-04-19T01:46:44.070 INFO:teuthology.orchestra.run.smithi062.stdout: "election_strategy": 1, 2024-04-19T01:46:44.070 INFO:teuthology.orchestra.run.smithi062.stdout: "disallowed_leaders: ": "", 2024-04-19T01:46:44.070 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false, 2024-04-19T01:46:44.070 INFO:teuthology.orchestra.run.smithi062.stdout: "tiebreaker_mon": "", 2024-04-19T01:46:44.070 INFO:teuthology.orchestra.run.smithi062.stdout: "removed_ranks: ": "", 2024-04-19T01:46:44.070 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:46:44.070 INFO:teuthology.orchestra.run.smithi062.stdout: "persistent": [ 2024-04-19T01:46:44.070 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:46:44.070 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:46:44.070 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:46:44.070 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:46:44.070 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:46:44.071 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:46:44.071 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:46:44.071 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:46:44.071 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:46:44.071 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:46:44.071 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:46:44.071 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:46:44.071 INFO:teuthology.orchestra.run.smithi062.stdout: "optional": [] 2024-04-19T01:46:44.071 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:44.071 INFO:teuthology.orchestra.run.smithi062.stdout: "mons": [ 2024-04-19T01:46:44.071 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:44.071 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 0, 2024-04-19T01:46:44.071 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "a", 2024-04-19T01:46:44.071 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:46:44.072 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:46:44.072 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:44.072 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:46:44.072 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3300", 2024-04-19T01:46:44.072 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:44.072 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:44.072 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:44.072 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:46:44.072 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789", 2024-04-19T01:46:44.072 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:44.072 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:44.072 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:44.072 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:44.072 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789/0", 2024-04-19T01:46:44.072 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6789/0", 2024-04-19T01:46:44.073 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:46:44.073 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:46:44.073 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:46:44.073 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:44.073 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:44.073 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 1, 2024-04-19T01:46:44.073 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "b", 2024-04-19T01:46:44.073 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:46:44.073 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:46:44.073 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:44.073 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:46:44.073 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:3300", 2024-04-19T01:46:44.073 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:44.073 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:44.074 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:44.074 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:46:44.074 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789", 2024-04-19T01:46:44.074 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:44.074 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:44.074 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:44.074 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:44.074 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789/0", 2024-04-19T01:46:44.074 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.71:6789/0", 2024-04-19T01:46:44.074 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:46:44.074 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:46:44.074 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:46:44.074 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:44.074 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:44.075 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 2, 2024-04-19T01:46:44.075 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "c", 2024-04-19T01:46:44.075 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:46:44.075 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:46:44.075 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:44.075 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:46:44.075 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3301", 2024-04-19T01:46:44.075 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:44.075 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:44.075 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:44.075 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:46:44.075 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790", 2024-04-19T01:46:44.075 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:44.075 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:44.075 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:44.075 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:44.076 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790/0", 2024-04-19T01:46:44.076 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6790/0", 2024-04-19T01:46:44.076 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:46:44.076 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:46:44.076 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:46:44.076 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:44.076 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:44.076 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:44.076 INFO:teuthology.orchestra.run.smithi062.stdout: "feature_map": { 2024-04-19T01:46:44.076 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": [ 2024-04-19T01:46:44.076 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:44.076 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:46:44.076 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:46:44.076 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:46:44.076 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:44.077 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:46:44.077 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": [ 2024-04-19T01:46:44.077 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:44.077 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:46:44.077 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:46:44.077 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:46:44.077 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:44.077 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:46:44.077 INFO:teuthology.orchestra.run.smithi062.stdout: "client": [ 2024-04-19T01:46:44.077 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:44.077 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:46:44.077 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:46:44.077 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 3 2024-04-19T01:46:44.077 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:44.077 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:44.077 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:44.077 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false 2024-04-19T01:46:44.078 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-19T01:46:44.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.063+0000 7fa48b4e7640 1 -- 172.21.15.62:0/4025171904 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa4680364a0 msgr2=0x7fa468038960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:44.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.063+0000 7fa48b4e7640 1 --2- 172.21.15.62:0/4025171904 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa4680364a0 0x7fa468038960 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa4841235c0 tx=0x7fa474009040 comp rx=0 tx=0).stop 2024-04-19T01:46:44.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.063+0000 7fa48b4e7640 10 monclient: shutdown 2024-04-19T01:46:44.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.063+0000 7fa48b4e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:46:44.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.063+0000 7fa48b4e7640 1 -- 172.21.15.62:0/4025171904 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4840a52d0 msgr2=0x7fa484137880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:44.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.063+0000 7fa489a84640 1 -- 172.21.15.62:0/4025171904 reap_dead start 2024-04-19T01:46:44.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.063+0000 7fa48b4e7640 1 --2- 172.21.15.62:0/4025171904 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4840a52d0 0x7fa484137880 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa47c01c6f0 tx=0x7fa47c019660 comp rx=0 tx=0).stop 2024-04-19T01:46:44.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.063+0000 7fa48b4e7640 1 -- 172.21.15.62:0/4025171904 shutdown_connections 2024-04-19T01:46:44.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.063+0000 7fa48b4e7640 1 --2- 172.21.15.62:0/4025171904 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4840a52d0 0x7fa484137880 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:44.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.063+0000 7fa48b4e7640 1 -- 172.21.15.62:0/4025171904 >> 172.21.15.62:0/4025171904 conn(0x7fa4840a23d0 msgr2=0x7fa484138410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:46:44.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.063+0000 7fa48b4e7640 1 -- 172.21.15.62:0/4025171904 shutdown_connections 2024-04-19T01:46:44.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.063+0000 7fa48b4e7640 1 -- 172.21.15.62:0/4025171904 wait complete. 2024-04-19T01:46:44.080 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-19T01:46:44.080 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-19T01:46:44.080 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-19T01:46:44.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f415f139640 10 monclient: get_monmap_and_config 2024-04-19T01:46:44.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f415f139640 10 monclient: build_initial_monmap 2024-04-19T01:46:44.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f415f139640 10 monclient: monmap: 2024-04-19T01:46:44.196 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:46:44.196 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:44.197 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:46:44.202109+0000 2024-04-19T01:46:44.197 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:46:44.202109+0000 2024-04-19T01:46:44.197 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:46:44.197 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:44.197 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:46:44.197 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:46:44.197 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:46:44.197 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:46:44.197 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:46:44.197 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:44.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f415f139640 1 Processor -- start 2024-04-19T01:46:44.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f415f139640 1 -- start start 2024-04-19T01:46:44.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f415f139640 10 monclient: init 2024-04-19T01:46:44.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f415f139640 10 monclient: _reopen_session rank -1 2024-04-19T01:46:44.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f415f139640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-04-19T01:46:44.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f415f139640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f41580a47e0 addr v1:172.21.15.71:6789/0 2024-04-19T01:46:44.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f415f139640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f41580a4bc0 0x7f41580a5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:44.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f415f139640 10 monclient(hunting): picked mon.noname-a con 0x7f41580a4bc0 addr v2:172.21.15.62:3300/0 2024-04-19T01:46:44.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f415f139640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4158139560 addr v1:172.21.15.62:6789/0 2024-04-19T01:46:44.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f415f139640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f415809fa60 con 0x7f4158139560 2024-04-19T01:46:44.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f415f139640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f415809ec50 con 0x7f41580a47e0 2024-04-19T01:46:44.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f415f139640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:44.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f415f139640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f415809e690 con 0x7f41580a4bc0 2024-04-19T01:46:44.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f415f139640 10 monclient(hunting): _renew_subs 2024-04-19T01:46:44.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f415f139640 10 monclient(hunting): authenticate will time out at 1190.021648s 2024-04-19T01:46:44.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f4157fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f41580a4bc0 0x7f41580a5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:44.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f4157fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f41580a4bc0 0x7f41580a5010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57216/0 (socket says 172.21.15.62:57216) 2024-04-19T01:46:44.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f415d6d6640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f4158139560 0x7f415813b950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:60084/0 (socket says 172.21.15.62:60084) 2024-04-19T01:46:44.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f4157fff640 1 -- 172.21.15.62:0/2525779392 learned_addr learned my addr 172.21.15.62:0/2525779392 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:46:44.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f4157fff640 10 monclient(hunting): get_auth_request con 0x7f41580a4bc0 auth_method 0 2024-04-19T01:46:44.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f4157fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:44.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f4157fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:44.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f4157fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:44.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f4157fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:44.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f4157fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:44.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f4157fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:44.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f4157fff640 10 monclient(hunting): handle_auth_done global_id 14727 payload 274 2024-04-19T01:46:44.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f4157fff640 1 -- 172.21.15.62:0/2525779392 >> v1:172.21.15.71:6789/0 conn(0x7f41580a47e0 legacy=0x7f415809f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:46:44.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f4157fff640 1 -- 172.21.15.62:0/2525779392 >> v1:172.21.15.62:6789/0 conn(0x7f4158139560 legacy=0x7f415813b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:46:44.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f4157fff640 10 monclient: _finish_hunting 0 2024-04-19T01:46:44.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f4157fff640 1 monclient: found mon.noname-a 2024-04-19T01:46:44.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f4157fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:46:44.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f4157fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:46:44.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.199+0000 7f4157fff640 1 -- 172.21.15.62:0/2525779392 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f415809e970 con 0x7f41580a4bc0 2024-04-19T01:46:44.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.203+0000 7f4157fff640 1 -- 172.21.15.62:0/2525779392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41580a4bc0 msgr2=0x7f41580a5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:44.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.203+0000 7f4157fff640 1 --2- 172.21.15.62:0/2525779392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41580a4bc0 0x7f41580a5010 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f414c0098a0 tx=0x7f414c01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2377d5c09305f2d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:44.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.203+0000 7f41577fe640 1 -- 172.21.15.62:0/2525779392 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c0b1070 con 0x7f41580a4bc0 2024-04-19T01:46:44.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.203+0000 7f41577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:44.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.203+0000 7f41577fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:46:44.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.203+0000 7f41577fe640 10 monclient: dump: 2024-04-19T01:46:44.203 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:44.203 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:44.203 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:44.203 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:44.203 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:44.203 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:44.203 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:44.203 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:44.204 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:44.204 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:44.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.203+0000 7f41577fe640 10 monclient: _finish_auth 0 2024-04-19T01:46:44.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.203+0000 7f41577fe640 10 monclient: _check_auth_tickets 2024-04-19T01:46:44.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.203+0000 7f41577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:46:44.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.203+0000 7f41577fe640 1 -- 172.21.15.62:0/2525779392 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f414c0ac600 con 0x7f41580a4bc0 2024-04-19T01:46:44.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.203+0000 7f415f139640 5 monclient: authenticate success, global_id 14727 2024-04-19T01:46:44.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.203+0000 7f415f139640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:46:44.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.203+0000 7f41577fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:46:44.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.203+0000 7f415f139640 10 monclient: get_monmap_and_config success 2024-04-19T01:46:44.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.203+0000 7f415f139640 4 set_mon_vals no callback set 2024-04-19T01:46:44.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.203+0000 7f41577fe640 1 -- 172.21.15.62:0/2525779392 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c0ac9d0 con 0x7f41580a4bc0 2024-04-19T01:46:44.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.203+0000 7f41577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:44.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.203+0000 7f41577fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:46:44.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.203+0000 7f41577fe640 10 monclient: dump: 2024-04-19T01:46:44.205 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:44.205 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:44.205 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:44.205 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:44.205 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:44.205 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:44.205 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:44.205 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:44.205 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:44.205 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:44.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.203+0000 7f415f139640 10 monclient: shutdown 2024-04-19T01:46:44.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.203+0000 7f415f139640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:46:44.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.203+0000 7f415f139640 1 -- 172.21.15.62:0/2525779392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41580a4bc0 msgr2=0x7f41580a5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:44.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.203+0000 7f415f139640 1 --2- 172.21.15.62:0/2525779392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41580a4bc0 0x7f41580a5010 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f414c0098a0 tx=0x7f414c01c400 comp rx=0 tx=0).stop 2024-04-19T01:46:44.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.203+0000 7f415f139640 1 -- 172.21.15.62:0/2525779392 shutdown_connections 2024-04-19T01:46:44.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.203+0000 7f415f139640 1 --2- 172.21.15.62:0/2525779392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41580a4bc0 0x7f41580a5010 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:44.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.203+0000 7f415f139640 1 -- 172.21.15.62:0/2525779392 >> 172.21.15.62:0/2525779392 conn(0x7f41580a23d0 msgr2=0x7f41580a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:46:44.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.203+0000 7f415f139640 1 -- 172.21.15.62:0/2525779392 shutdown_connections 2024-04-19T01:46:44.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.203+0000 7f415f139640 1 -- 172.21.15.62:0/2525779392 wait complete. 2024-04-19T01:46:44.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.203+0000 7f415f139640 10 monclient: build_initial_monmap 2024-04-19T01:46:44.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.203+0000 7f415f139640 10 monclient: monmap: 2024-04-19T01:46:44.211 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:46:44.211 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:46:44.211 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:46:44.211 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:46:44.211 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:46:44.211 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:44.211 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:46:44.211 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:46:44.211 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:46:44.211 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:44.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415f139640 1 Processor -- start 2024-04-19T01:46:44.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415f139640 1 -- start start 2024-04-19T01:46:44.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415f139640 10 monclient: init 2024-04-19T01:46:44.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415f139640 10 monclient: _reopen_session rank -1 2024-04-19T01:46:44.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415f139640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-19T01:46:44.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415f139640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41580a47e0 0x7f41581cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:44.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415f139640 10 monclient(hunting): picked mon.noname-a con 0x7f41580a47e0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:46:44.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415f139640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41580a4bc0 0x7f4158135020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:44.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415f139640 10 monclient(hunting): picked mon.noname-b con 0x7f41580a4bc0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:46:44.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415f139640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4158139560 0x7f415811bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:44.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415f139640 10 monclient(hunting): picked mon.noname-c con 0x7f4158139560 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:46:44.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415f139640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:44.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415f139640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f415813fd10 con 0x7f41580a47e0 2024-04-19T01:46:44.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415f139640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:44.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415f139640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f415813fb90 con 0x7f41580a4bc0 2024-04-19T01:46:44.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415f139640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:44.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415f139640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f415813fe90 con 0x7f4158139560 2024-04-19T01:46:44.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415f139640 10 monclient(hunting): _renew_subs 2024-04-19T01:46:44.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415f139640 10 monclient(hunting): authenticate will time out at 1190.028820s 2024-04-19T01:46:44.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415ced5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41580a47e0 0x7f41581cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:44.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415ced5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41580a47e0 0x7f41581cd1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57230/0 (socket says 172.21.15.62:57230) 2024-04-19T01:46:44.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415d6d6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4158139560 0x7f415811bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:44.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415ced5640 1 -- 172.21.15.62:0/4191147850 learned_addr learned my addr 172.21.15.62:0/4191147850 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:46:44.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415ced5640 10 monclient(hunting): get_auth_request con 0x7f41580a47e0 auth_method 0 2024-04-19T01:46:44.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415ced5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:44.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415ced5640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:44.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415ced5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:44.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415d6d6640 10 monclient(hunting): get_auth_request con 0x7f4158139560 auth_method 0 2024-04-19T01:46:44.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415d6d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:44.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415d6d6640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:44.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415d6d6640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:44.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415ced5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:44.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415ced5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:44.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415ced5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:44.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415d6d6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:44.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415d6d6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:44.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415d6d6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:44.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f4157fff640 1 --2- 172.21.15.62:0/4191147850 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41580a4bc0 0x7f4158135020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:44.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f4157fff640 10 monclient(hunting): get_auth_request con 0x7f41580a4bc0 auth_method 0 2024-04-19T01:46:44.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f4157fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:44.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f4157fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:44.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f4157fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:44.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415ced5640 10 monclient(hunting): handle_auth_done global_id 14730 payload 824 2024-04-19T01:46:44.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415ced5640 1 -- 172.21.15.62:0/4191147850 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4158139560 msgr2=0x7f415811bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:44.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415ced5640 1 --2- 172.21.15.62:0/4191147850 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4158139560 0x7f415811bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:44.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415ced5640 1 -- 172.21.15.62:0/4191147850 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41580a4bc0 msgr2=0x7f4158135020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:44.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415ced5640 1 --2- 172.21.15.62:0/4191147850 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41580a4bc0 0x7f4158135020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:44.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415ced5640 10 monclient: _finish_hunting 0 2024-04-19T01:46:44.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415ced5640 1 monclient: found mon.noname-a 2024-04-19T01:46:44.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415ced5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:46:44.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415ced5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:46:44.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415ced5640 1 -- 172.21.15.62:0/4191147850 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f415811f9d0 con 0x7f41580a47e0 2024-04-19T01:46:44.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415d6d6640 1 --2- 172.21.15.62:0/4191147850 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4158139560 0x7f415811bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:46:44.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415ced5640 1 -- 172.21.15.62:0/4191147850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41580a47e0 msgr2=0x7f41581cd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:44.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415ced5640 1 --2- 172.21.15.62:0/4191147850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41580a47e0 0x7f41581cd1a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f414800c480 tx=0x7f414800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:44.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f41557fa640 1 -- 172.21.15.62:0/4191147850 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4148012a40 con 0x7f41580a47e0 2024-04-19T01:46:44.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f41557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:44.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f41557fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:46:44.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f41557fa640 10 monclient: dump: 2024-04-19T01:46:44.215 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:44.216 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:44.216 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:44.216 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:44.216 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:44.216 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:44.216 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:44.216 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:44.216 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:44.216 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:44.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f41557fa640 10 monclient: _finish_auth 0 2024-04-19T01:46:44.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f41557fa640 10 monclient: _check_auth_tickets 2024-04-19T01:46:44.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f41557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:46:44.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415f139640 5 monclient: authenticate success, global_id 14730 2024-04-19T01:46:44.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415f139640 10 monclient: _renew_subs 2024-04-19T01:46:44.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f41557fa640 1 -- 172.21.15.62:0/4191147850 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4148017070 con 0x7f41580a47e0 2024-04-19T01:46:44.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415f139640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:46:44.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415f139640 1 -- 172.21.15.62:0/4191147850 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41581ce090 con 0x7f41580a47e0 2024-04-19T01:46:44.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f41557fa640 10 monclient: handle_config config(0 keys) 2024-04-19T01:46:44.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f41557fa640 1 -- 172.21.15.62:0/4191147850 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4148012e40 con 0x7f41580a47e0 2024-04-19T01:46:44.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f41557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:44.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f41557fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:46:44.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f41557fa640 10 monclient: dump: 2024-04-19T01:46:44.217 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:44.217 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:44.217 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:44.217 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:44.217 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:44.217 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:44.217 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:44.217 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:44.218 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:44.218 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:44.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f4156ffd640 4 set_mon_vals no callback set 2024-04-19T01:46:44.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415f139640 10 monclient: _renew_subs 2024-04-19T01:46:44.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415f139640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:46:44.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.207+0000 7f415f139640 1 -- 172.21.15.62:0/4191147850 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f415811e3f0 con 0x7f41580a47e0 2024-04-19T01:46:44.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.211+0000 7f415f139640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:44.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.211+0000 7f415f139640 1 --2- 172.21.15.62:0/4191147850 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f415811c460 0x7f415811c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:44.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.211+0000 7f415f139640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:44.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.211+0000 7f415f139640 1 -- 172.21.15.62:0/4191147850 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f415811e000 con 0x7f415811c460 2024-04-19T01:46:44.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.211+0000 7f4157fff640 1 --2- 172.21.15.62:0/4191147850 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f415811c460 0x7f415811c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:44.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.211+0000 7f4157fff640 10 monclient: get_auth_request con 0x7f415811c460 auth_method 0 2024-04-19T01:46:44.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.211+0000 7f4157fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:44.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.211+0000 7f4157fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:44.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.211+0000 7f4157fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:44.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.211+0000 7f41557fa640 1 -- 172.21.15.62:0/4191147850 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f414801c460 con 0x7f41580a47e0 2024-04-19T01:46:44.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.211+0000 7f4157fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:44.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.211+0000 7f4157fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:44.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.211+0000 7f4157fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:44.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.211+0000 7f4157fff640 10 monclient(hunting): handle_auth_done global_id 5264 payload 824 2024-04-19T01:46:44.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.211+0000 7f4157fff640 1 -- 172.21.15.62:0/4191147850 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f415811cd80 con 0x7f415811c460 2024-04-19T01:46:44.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.211+0000 7f41557fa640 1 --2- 172.21.15.62:0/4191147850 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4144036450 0x7f4144038910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:44.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.211+0000 7f41557fa640 1 -- 172.21.15.62:0/4191147850 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(218..218 src has 108..218) ==== 4738+0+0 (secure 0 0 0) 0x7f41480492d0 con 0x7f41580a47e0 2024-04-19T01:46:44.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.211+0000 7f415d6d6640 1 --2- 172.21.15.62:0/4191147850 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4144036450 0x7f4144038910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:44.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.211+0000 7f415d6d6640 10 monclient: get_auth_request con 0x7f4144036450 auth_method 0 2024-04-19T01:46:44.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.211+0000 7f4157fff640 1 -- 172.21.15.62:0/4191147850 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f415811c460 msgr2=0x7f415811c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:44.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.211+0000 7f4157fff640 1 --2- 172.21.15.62:0/4191147850 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f415811c460 0x7f415811c840 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f415809dcf0 tx=0x7f414c0191a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:44.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.211+0000 7f41557fa640 1 -- 172.21.15.62:0/4191147850 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c0bc070 con 0x7f415811c460 2024-04-19T01:46:44.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.211+0000 7f41557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:44.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.211+0000 7f41557fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:46:44.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.211+0000 7f41557fa640 10 monclient: dump: 2024-04-19T01:46:44.221 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:44.221 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:44.222 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:44.222 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:44.222 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:44.222 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:44.222 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:44.222 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:44.222 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:44.222 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:44.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.211+0000 7f41557fa640 1 -- 172.21.15.62:0/4191147850 <== mon.2 v2:172.21.15.62:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f414c0b8460 con 0x7f415811c460 2024-04-19T01:46:44.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.211+0000 7f41557fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:44.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.211+0000 7f41557fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:46:44.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.211+0000 7f41557fa640 1 -- 172.21.15.62:0/4191147850 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f415811c460 msgr2=0x7f415811c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:44.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.211+0000 7f41557fa640 1 --2- 172.21.15.62:0/4191147850 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f415811c460 0x7f415811c840 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f415809dcf0 tx=0x7f414c0191a0 comp rx=0 tx=0).stop 2024-04-19T01:46:44.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.211+0000 7f41557fa640 1 -- 172.21.15.62:0/4191147850 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f415811c460 msgr2=0x7f415811c840 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:46:44.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.211+0000 7f41557fa640 1 --2- 172.21.15.62:0/4191147850 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f415811c460 0x7f415811c840 secure :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f415809dcf0 tx=0x7f414c0191a0 comp rx=0 tx=0).stop 2024-04-19T01:46:44.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.215+0000 7f415d6d6640 1 --2- 172.21.15.62:0/4191147850 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4144036450 0x7f4144038910 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f415809dd50 tx=0x7f4150004f40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:44.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.219+0000 7f415f139640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:46:44.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.219+0000 7f415f139640 1 --2- 172.21.15.62:0/4191147850 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f415806d960 0x7f415806dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:44.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.219+0000 7f415f139640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:44.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.219+0000 7f415f139640 1 -- 172.21.15.62:0/4191147850 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f41580a1d20 con 0x7f415806d960 2024-04-19T01:46:44.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.219+0000 7f4157fff640 1 --2- 172.21.15.62:0/4191147850 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f415806d960 0x7f415806dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:44.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.219+0000 7f4157fff640 10 monclient: get_auth_request con 0x7f415806d960 auth_method 0 2024-04-19T01:46:44.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.219+0000 7f4157fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:44.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.219+0000 7f4157fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:44.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.219+0000 7f4157fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:44.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.219+0000 7f4157fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:44.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.219+0000 7f4157fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:44.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.219+0000 7f4157fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:44.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.219+0000 7f4157fff640 10 monclient(hunting): handle_auth_done global_id 5267 payload 824 2024-04-19T01:46:44.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.219+0000 7f4157fff640 1 -- 172.21.15.62:0/4191147850 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f415811d670 con 0x7f415806d960 2024-04-19T01:46:44.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.223+0000 7f4157fff640 1 -- 172.21.15.62:0/4191147850 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f415806d960 msgr2=0x7f415806dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:44.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.223+0000 7f4157fff640 1 --2- 172.21.15.62:0/4191147850 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f415806d960 0x7f415806dd70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f414c01aab0 tx=0x7f414c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:44.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.223+0000 7f41557fa640 1 -- 172.21.15.62:0/4191147850 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c0b1040 con 0x7f415806d960 2024-04-19T01:46:44.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.223+0000 7f41557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:44.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.223+0000 7f41557fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:46:44.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.223+0000 7f41557fa640 10 monclient: dump: 2024-04-19T01:46:44.225 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:44.225 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:44.225 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:44.225 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:44.225 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:44.225 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:44.225 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:44.225 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:44.225 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:44.226 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:44.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.223+0000 7f41557fa640 1 -- 172.21.15.62:0/4191147850 <== mon.2 v2:172.21.15.62:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f414c0b6430 con 0x7f415806d960 2024-04-19T01:46:44.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.223+0000 7f41557fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:46:44.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.223+0000 7f41557fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:46:44.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.223+0000 7f41557fa640 1 -- 172.21.15.62:0/4191147850 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f415806d960 msgr2=0x7f415806dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:44.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.223+0000 7f41557fa640 1 --2- 172.21.15.62:0/4191147850 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f415806d960 0x7f415806dd70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f414c01aab0 tx=0x7f414c008040 comp rx=0 tx=0).stop 2024-04-19T01:46:44.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.223+0000 7f41557fa640 1 -- 172.21.15.62:0/4191147850 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f415806d960 msgr2=0x7f415806dd70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:46:44.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.223+0000 7f41557fa640 1 --2- 172.21.15.62:0/4191147850 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f415806d960 0x7f415806dd70 secure :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f414c01aab0 tx=0x7f414c008040 comp rx=0 tx=0).stop 2024-04-19T01:46:44.226 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-19T01:46:44.226 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "c", 2024-04-19T01:46:44.226 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 2, 2024-04-19T01:46:44.227 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "peon", 2024-04-19T01:46:44.227 INFO:teuthology.orchestra.run.smithi062.stdout: "election_epoch": 34, 2024-04-19T01:46:44.227 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 280428, 2024-04-19T01:46:44.227 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum": [ 2024-04-19T01:46:44.227 INFO:teuthology.orchestra.run.smithi062.stdout: 0, 2024-04-19T01:46:44.227 INFO:teuthology.orchestra.run.smithi062.stdout: 1, 2024-04-19T01:46:44.227 INFO:teuthology.orchestra.run.smithi062.stdout: 2 2024-04-19T01:46:44.227 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:46:44.227 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_age": 0, 2024-04-19T01:46:44.227 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:46:44.227 INFO:teuthology.orchestra.run.smithi062.stdout: "required_con": "2450521708007866372", 2024-04-19T01:46:44.227 INFO:teuthology.orchestra.run.smithi062.stdout: "required_mon": [ 2024-04-19T01:46:44.227 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:46:44.227 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:46:44.227 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:46:44.228 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:46:44.228 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:46:44.228 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:46:44.228 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:46:44.228 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:46:44.228 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:46:44.228 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:46:44.228 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:46:44.228 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:46:44.228 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:46:44.228 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_mon": [ 2024-04-19T01:46:44.228 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:46:44.228 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:46:44.228 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:46:44.228 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:46:44.229 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:46:44.229 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:46:44.229 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:46:44.229 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:46:44.229 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:46:44.229 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:46:44.229 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:46:44.229 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:44.229 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:44.229 INFO:teuthology.orchestra.run.smithi062.stdout: "outside_quorum": [], 2024-04-19T01:46:44.229 INFO:teuthology.orchestra.run.smithi062.stdout: "extra_probe_peers": [], 2024-04-19T01:46:44.229 INFO:teuthology.orchestra.run.smithi062.stdout: "sync_provider": [], 2024-04-19T01:46:44.229 INFO:teuthology.orchestra.run.smithi062.stdout: "monmap": { 2024-04-19T01:46:44.229 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 1, 2024-04-19T01:46:44.229 INFO:teuthology.orchestra.run.smithi062.stdout: "fsid": "6466e4b6-2520-485f-95b3-9401915fbbad", 2024-04-19T01:46:44.230 INFO:teuthology.orchestra.run.smithi062.stdout: "modified": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:46:44.230 INFO:teuthology.orchestra.run.smithi062.stdout: "created": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:46:44.230 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release": 19, 2024-04-19T01:46:44.230 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release_name": "squid", 2024-04-19T01:46:44.230 INFO:teuthology.orchestra.run.smithi062.stdout: "election_strategy": 1, 2024-04-19T01:46:44.230 INFO:teuthology.orchestra.run.smithi062.stdout: "disallowed_leaders: ": "", 2024-04-19T01:46:44.230 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false, 2024-04-19T01:46:44.230 INFO:teuthology.orchestra.run.smithi062.stdout: "tiebreaker_mon": "", 2024-04-19T01:46:44.230 INFO:teuthology.orchestra.run.smithi062.stdout: "removed_ranks: ": "", 2024-04-19T01:46:44.230 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:46:44.230 INFO:teuthology.orchestra.run.smithi062.stdout: "persistent": [ 2024-04-19T01:46:44.230 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:46:44.230 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:46:44.230 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:46:44.230 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:46:44.231 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:46:44.231 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:46:44.231 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:46:44.231 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:46:44.231 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:46:44.231 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:46:44.231 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:46:44.231 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:46:44.231 INFO:teuthology.orchestra.run.smithi062.stdout: "optional": [] 2024-04-19T01:46:44.231 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:44.231 INFO:teuthology.orchestra.run.smithi062.stdout: "mons": [ 2024-04-19T01:46:44.231 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:44.231 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 0, 2024-04-19T01:46:44.231 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "a", 2024-04-19T01:46:44.231 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:46:44.232 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:46:44.232 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:44.232 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:46:44.232 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3300", 2024-04-19T01:46:44.232 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:44.232 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:44.232 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:44.232 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:46:44.232 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789", 2024-04-19T01:46:44.232 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:44.232 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:44.232 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:44.232 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:44.232 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789/0", 2024-04-19T01:46:44.232 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6789/0", 2024-04-19T01:46:44.232 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:46:44.232 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:46:44.233 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:46:44.233 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:44.233 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:44.233 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 1, 2024-04-19T01:46:44.233 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "b", 2024-04-19T01:46:44.233 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:46:44.233 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:46:44.233 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:44.233 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:46:44.233 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:3300", 2024-04-19T01:46:44.233 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:44.233 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:44.233 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:44.233 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:46:44.233 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789", 2024-04-19T01:46:44.233 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:44.234 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:44.234 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:44.234 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:44.234 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789/0", 2024-04-19T01:46:44.234 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.71:6789/0", 2024-04-19T01:46:44.234 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:46:44.234 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:46:44.234 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:46:44.234 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:44.234 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:44.234 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 2, 2024-04-19T01:46:44.234 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "c", 2024-04-19T01:46:44.234 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:46:44.234 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:46:44.234 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:44.235 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:46:44.235 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3301", 2024-04-19T01:46:44.235 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:44.235 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:44.235 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:44.235 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:46:44.235 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790", 2024-04-19T01:46:44.235 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:44.235 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:44.235 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:44.235 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:44.235 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790/0", 2024-04-19T01:46:44.235 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6790/0", 2024-04-19T01:46:44.235 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:46:44.235 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:46:44.235 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:46:44.236 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:44.236 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:44.236 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:44.236 INFO:teuthology.orchestra.run.smithi062.stdout: "feature_map": { 2024-04-19T01:46:44.236 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": [ 2024-04-19T01:46:44.236 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:44.236 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:46:44.236 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:46:44.236 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:46:44.236 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:44.236 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:46:44.236 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": [ 2024-04-19T01:46:44.236 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:44.236 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:46:44.236 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:46:44.236 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 5 2024-04-19T01:46:44.237 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:44.237 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:46:44.237 INFO:teuthology.orchestra.run.smithi062.stdout: "client": [ 2024-04-19T01:46:44.237 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:44.237 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:46:44.237 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:46:44.237 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 3 2024-04-19T01:46:44.237 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:44.237 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:46:44.237 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": [ 2024-04-19T01:46:44.237 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:44.237 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:46:44.237 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:46:44.237 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:46:44.237 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:44.237 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:44.238 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:44.238 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false 2024-04-19T01:46:44.238 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-19T01:46:44.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.223+0000 7f415f139640 1 -- 172.21.15.62:0/4191147850 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4144036450 msgr2=0x7f4144038910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:44.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.223+0000 7f415f139640 1 --2- 172.21.15.62:0/4191147850 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4144036450 0x7f4144038910 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f415809dd50 tx=0x7f4150004f40 comp rx=0 tx=0).stop 2024-04-19T01:46:44.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.223+0000 7f415f139640 10 monclient: shutdown 2024-04-19T01:46:44.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.223+0000 7f415f139640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:46:44.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.223+0000 7f415f139640 1 -- 172.21.15.62:0/4191147850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41580a47e0 msgr2=0x7f41581cd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:44.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.223+0000 7f415f139640 1 --2- 172.21.15.62:0/4191147850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41580a47e0 0x7f41581cd1a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f414800c480 tx=0x7f414800c940 comp rx=0 tx=0).stop 2024-04-19T01:46:44.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.223+0000 7f415d6d6640 1 -- 172.21.15.62:0/4191147850 reap_dead start 2024-04-19T01:46:44.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.223+0000 7f415f139640 1 -- 172.21.15.62:0/4191147850 shutdown_connections 2024-04-19T01:46:44.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.223+0000 7f415f139640 1 --2- 172.21.15.62:0/4191147850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41580a47e0 0x7f41581cd1a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:44.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.223+0000 7f415f139640 1 -- 172.21.15.62:0/4191147850 >> 172.21.15.62:0/4191147850 conn(0x7f41580a23d0 msgr2=0x7f415809ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:46:44.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.223+0000 7f415f139640 1 -- 172.21.15.62:0/4191147850 shutdown_connections 2024-04-19T01:46:44.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.223+0000 7f415f139640 1 -- 172.21.15.62:0/4191147850 wait complete. 2024-04-19T01:46:44.240 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30398]: starting. 2024-04-19T01:46:44.240 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30398]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-30396 2024-04-19T01:46:44.240 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30398]: shutting down. 2024-04-19T01:46:44.240 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-19T01:46:44.240 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-19T01:46:44.241 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T01:46:44.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.355+0000 7f8e50be6640 10 monclient: get_monmap_and_config 2024-04-19T01:46:44.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.355+0000 7f8e50be6640 10 monclient: build_initial_monmap 2024-04-19T01:46:44.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.355+0000 7f8e50be6640 10 monclient: monmap: 2024-04-19T01:46:44.355 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:46:44.355 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:44.355 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:46:44.360495+0000 2024-04-19T01:46:44.355 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:46:44.360495+0000 2024-04-19T01:46:44.355 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:46:44.355 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:44.355 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:46:44.355 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:46:44.355 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:46:44.355 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:46:44.355 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:46:44.355 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:44.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.355+0000 7f8e50be6640 1 Processor -- start 2024-04-19T01:46:44.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.355+0000 7f8e50be6640 1 -- start start 2024-04-19T01:46:44.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.355+0000 7f8e50be6640 10 monclient: init 2024-04-19T01:46:44.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.355+0000 7f8e50be6640 10 monclient: _reopen_session rank -1 2024-04-19T01:46:44.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.355+0000 7f8e50be6640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-04-19T01:46:44.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.355+0000 7f8e50be6640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8e4c137780 0x7f8e4c137b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:44.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.355+0000 7f8e50be6640 10 monclient(hunting): picked mon.noname-a con 0x7f8e4c137780 addr v2:172.21.15.62:3300/0 2024-04-19T01:46:44.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.355+0000 7f8e50be6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8e4c1380a0 0x7f8e4c0a45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:44.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.355+0000 7f8e50be6640 10 monclient(hunting): picked mon.noname-b con 0x7f8e4c1380a0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:46:44.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.355+0000 7f8e50be6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8e4c0a60b0 addr v1:172.21.15.71:6789/0 2024-04-19T01:46:44.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.355+0000 7f8e50be6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e4c09c2e0 con 0x7f8e4c0a60b0 2024-04-19T01:46:44.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.355+0000 7f8e50be6640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:44.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.355+0000 7f8e50be6640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f8e4c09bd20 con 0x7f8e4c137780 2024-04-19T01:46:44.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.355+0000 7f8e50be6640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:44.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.355+0000 7f8e50be6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f8e4c09c000 con 0x7f8e4c1380a0 2024-04-19T01:46:44.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.355+0000 7f8e50be6640 10 monclient(hunting): _renew_subs 2024-04-19T01:46:44.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.355+0000 7f8e50be6640 10 monclient(hunting): authenticate will time out at 1190.179768s 2024-04-19T01:46:44.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.355+0000 7f8e4a59c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8e4c137780 0x7f8e4c137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:44.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.355+0000 7f8e49d9b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8e4c1380a0 0x7f8e4c0a45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:44.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.355+0000 7f8e4a59c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8e4c137780 0x7f8e4c137b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57244/0 (socket says 172.21.15.62:57244) 2024-04-19T01:46:44.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.359+0000 7f8e4a59c640 1 -- 172.21.15.62:0/283130151 learned_addr learned my addr 172.21.15.62:0/283130151 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:46:44.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.359+0000 7f8e4a59c640 10 monclient(hunting): get_auth_request con 0x7f8e4c137780 auth_method 0 2024-04-19T01:46:44.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.359+0000 7f8e4a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:44.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.359+0000 7f8e4a59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:44.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.359+0000 7f8e4a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:44.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.359+0000 7f8e49d9b640 10 monclient(hunting): get_auth_request con 0x7f8e4c1380a0 auth_method 0 2024-04-19T01:46:44.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.359+0000 7f8e49d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:44.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.359+0000 7f8e49d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:44.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.359+0000 7f8e49d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:44.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.359+0000 7f8e4a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:44.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.359+0000 7f8e4a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:44.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.359+0000 7f8e4a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:44.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.359+0000 7f8e49d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:44.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.359+0000 7f8e49d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:44.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.359+0000 7f8e49d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:44.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.359+0000 7f8e49d9b640 10 monclient(hunting): handle_auth_done global_id 5282 payload 274 2024-04-19T01:46:44.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.359+0000 7f8e49d9b640 1 -- 172.21.15.62:0/283130151 >> v2:172.21.15.62:3300/0 conn(0x7f8e4c137780 msgr2=0x7f8e4c137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:46:44.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.359+0000 7f8e49d9b640 1 --2- 172.21.15.62:0/283130151 >> v2:172.21.15.62:3300/0 conn(0x7f8e4c137780 0x7f8e4c137b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:44.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.359+0000 7f8e49d9b640 1 -- 172.21.15.62:0/283130151 >> v1:172.21.15.71:6789/0 conn(0x7f8e4c0a60b0 legacy=0x7f8e4c0a4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:46:44.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.359+0000 7f8e49d9b640 10 monclient: _finish_hunting 0 2024-04-19T01:46:44.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.359+0000 7f8e49d9b640 1 monclient: found mon.noname-b 2024-04-19T01:46:44.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.359+0000 7f8e49d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:46:44.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.359+0000 7f8e49d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:46:44.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.359+0000 7f8e49d9b640 1 -- 172.21.15.62:0/283130151 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e4c073210 con 0x7f8e4c1380a0 2024-04-19T01:46:44.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.359+0000 7f8e4a59c640 1 --2- 172.21.15.62:0/283130151 >> v2:172.21.15.62:3300/0 conn(0x7f8e4c137780 0x7f8e4c137b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:46:44.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e49d9b640 1 -- 172.21.15.62:0/283130151 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8e4c1380a0 msgr2=0x7f8e4c0a45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:44.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e49d9b640 1 --2- 172.21.15.62:0/283130151 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8e4c1380a0 0x7f8e4c0a45d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f8e400097c0 tx=0x7f8e4001c180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4048de765ee7f8a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:44.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4959a640 1 -- 172.21.15.62:0/283130151 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e400b0070 con 0x7f8e4c1380a0 2024-04-19T01:46:44.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:44.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:46:44.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4959a640 10 monclient: dump: 2024-04-19T01:46:44.362 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:44.362 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:44.362 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:44.362 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:44.362 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:44.363 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:44.363 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:44.363 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:44.363 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:44.363 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:44.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4959a640 10 monclient: _finish_auth 0 2024-04-19T01:46:44.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4959a640 10 monclient: _check_auth_tickets 2024-04-19T01:46:44.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:46:44.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4959a640 1 -- 172.21.15.62:0/283130151 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e400ab600 con 0x7f8e4c1380a0 2024-04-19T01:46:44.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4959a640 10 monclient: handle_config config(0 keys) 2024-04-19T01:46:44.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4959a640 1 -- 172.21.15.62:0/283130151 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e400ab9d0 con 0x7f8e4c1380a0 2024-04-19T01:46:44.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:44.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:46:44.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4959a640 10 monclient: dump: 2024-04-19T01:46:44.363 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:44.364 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:44.364 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:44.364 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:44.364 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:44.364 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:44.364 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:44.364 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:44.364 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:44.364 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:44.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 5 monclient: authenticate success, global_id 5282 2024-04-19T01:46:44.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 10 monclient: get_monmap_and_config success 2024-04-19T01:46:44.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 4 set_mon_vals no callback set 2024-04-19T01:46:44.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 10 monclient: shutdown 2024-04-19T01:46:44.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:46:44.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 1 -- 172.21.15.62:0/283130151 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8e4c1380a0 msgr2=0x7f8e4c0a45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:44.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 1 --2- 172.21.15.62:0/283130151 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8e4c1380a0 0x7f8e4c0a45d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f8e400097c0 tx=0x7f8e4001c180 comp rx=0 tx=0).stop 2024-04-19T01:46:44.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 1 -- 172.21.15.62:0/283130151 shutdown_connections 2024-04-19T01:46:44.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 1 --2- 172.21.15.62:0/283130151 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8e4c1380a0 0x7f8e4c0a45d0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:44.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 1 --2- 172.21.15.62:0/283130151 >> v2:172.21.15.62:3300/0 conn(0x7f8e4c137780 0x7f8e4c137b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:44.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 1 -- 172.21.15.62:0/283130151 >> 172.21.15.62:0/283130151 conn(0x7f8e4c0aa1d0 msgr2=0x7f8e4c0a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:46:44.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 1 -- 172.21.15.62:0/283130151 shutdown_connections 2024-04-19T01:46:44.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 1 -- 172.21.15.62:0/283130151 wait complete. 2024-04-19T01:46:44.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 10 monclient: build_initial_monmap 2024-04-19T01:46:44.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 10 monclient: monmap: 2024-04-19T01:46:44.365 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:46:44.365 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:46:44.365 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:46:44.365 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:46:44.365 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:46:44.365 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:44.365 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:46:44.365 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:46:44.365 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:46:44.366 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:44.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 1 Processor -- start 2024-04-19T01:46:44.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 1 -- start start 2024-04-19T01:46:44.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 10 monclient: init 2024-04-19T01:46:44.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 10 monclient: _reopen_session rank -1 2024-04-19T01:46:44.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-19T01:46:44.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8e4c0a60b0 0x7f8e4c1cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:44.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 10 monclient(hunting): picked mon.noname-c con 0x7f8e4c0a60b0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:46:44.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e4c137780 0x7f8e4c136360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:44.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 10 monclient(hunting): picked mon.noname-a con 0x7f8e4c137780 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:46:44.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c1380a0 0x7f8e4c13c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:44.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 10 monclient(hunting): picked mon.noname-b con 0x7f8e4c1380a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:46:44.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:44.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4c142170 con 0x7f8e4c137780 2024-04-19T01:46:44.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:44.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4c0a5430 con 0x7f8e4c1380a0 2024-04-19T01:46:44.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:44.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f8e4c1422f0 con 0x7f8e4c0a60b0 2024-04-19T01:46:44.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 10 monclient(hunting): _renew_subs 2024-04-19T01:46:44.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e50be6640 10 monclient(hunting): authenticate will time out at 1190.187204s 2024-04-19T01:46:44.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e49d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e4c137780 0x7f8e4c136360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:44.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4a59c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8e4c0a60b0 0x7f8e4c1cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:44.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4a59c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8e4c0a60b0 0x7f8e4c1cf5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:50768/0 (socket says 172.21.15.62:50768) 2024-04-19T01:46:44.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4a59c640 1 -- 172.21.15.62:0/2606783357 learned_addr learned my addr 172.21.15.62:0/2606783357 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:46:44.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4a59c640 10 monclient(hunting): get_auth_request con 0x7f8e4c0a60b0 auth_method 0 2024-04-19T01:46:44.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:44.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4a59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:44.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:44.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e49d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e4c137780 0x7f8e4c136360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57250/0 (socket says 172.21.15.62:57250) 2024-04-19T01:46:44.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e49d9b640 10 monclient(hunting): get_auth_request con 0x7f8e4c137780 auth_method 0 2024-04-19T01:46:44.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e49d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:44.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e49d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:44.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e49d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:44.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:44.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:44.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:44.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e49d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:44.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e49d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:44.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e49d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:44.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4ad9d640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c1380a0 0x7f8e4c13c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:44.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4a59c640 10 monclient(hunting): handle_auth_done global_id 5285 payload 824 2024-04-19T01:46:44.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4a59c640 1 -- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c1380a0 msgr2=0x7f8e4c13c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:44.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4a59c640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c1380a0 0x7f8e4c13c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:44.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4a59c640 1 -- 172.21.15.62:0/2606783357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e4c137780 msgr2=0x7f8e4c136360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:44.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4a59c640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e4c137780 0x7f8e4c136360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:44.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4a59c640 10 monclient: _finish_hunting 0 2024-04-19T01:46:44.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4a59c640 1 monclient: found mon.noname-c 2024-04-19T01:46:44.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:46:44.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:46:44.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4a59c640 1 -- 172.21.15.62:0/2606783357 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e4c120cf0 con 0x7f8e4c0a60b0 2024-04-19T01:46:44.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e49d9b640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e4c137780 0x7f8e4c136360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:46:44.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4ad9d640 10 monclient: get_auth_request con 0x7f8e4c1380a0 auth_method 0 2024-04-19T01:46:44.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.363+0000 7f8e4ad9d640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c1380a0 0x7f8e4c13c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:46:44.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e4a59c640 1 -- 172.21.15.62:0/2606783357 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8e4c0a60b0 msgr2=0x7f8e4c1cf5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:44.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e4a59c640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8e4c0a60b0 0x7f8e4c1cf5e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c00ea30 tx=0x7f8e3c00eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:44.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e3affd640 1 -- 172.21.15.62:0/2606783357 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c017070 con 0x7f8e4c0a60b0 2024-04-19T01:46:44.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:44.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e3affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:46:44.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e3affd640 10 monclient: dump: 2024-04-19T01:46:44.374 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:44.374 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:44.374 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:44.374 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:44.374 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:44.374 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:44.374 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:44.375 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:44.375 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:44.375 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:44.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e3affd640 10 monclient: _finish_auth 0 2024-04-19T01:46:44.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e3affd640 10 monclient: _check_auth_tickets 2024-04-19T01:46:44.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:46:44.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e3affd640 1 -- 172.21.15.62:0/2606783357 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e3c00ccd0 con 0x7f8e4c0a60b0 2024-04-19T01:46:44.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e50be6640 5 monclient: authenticate success, global_id 5285 2024-04-19T01:46:44.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e50be6640 10 monclient: _renew_subs 2024-04-19T01:46:44.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e50be6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:46:44.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e50be6640 1 -- 172.21.15.62:0/2606783357 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e4c0a3420 con 0x7f8e4c0a60b0 2024-04-19T01:46:44.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e3affd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:46:44.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e3affd640 1 -- 172.21.15.62:0/2606783357 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c013670 con 0x7f8e4c0a60b0 2024-04-19T01:46:44.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:44.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:46:44.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e3affd640 10 monclient: dump: 2024-04-19T01:46:44.376 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:44.376 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:44.376 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:44.376 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:44.376 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:44.376 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:44.376 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:44.376 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:44.376 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:44.376 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:44.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e48d99640 4 set_mon_vals no callback set 2024-04-19T01:46:44.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e50be6640 10 monclient: _renew_subs 2024-04-19T01:46:44.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e50be6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:46:44.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e50be6640 1 -- 172.21.15.62:0/2606783357 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e4c1d0460 con 0x7f8e4c0a60b0 2024-04-19T01:46:44.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e50be6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:44.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e50be6640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c13c780 0x7f8e4c13cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:44.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e50be6640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:44.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e50be6640 1 -- 172.21.15.62:0/2606783357 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4c120670 con 0x7f8e4c13c780 2024-04-19T01:46:44.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e3affd640 1 -- 172.21.15.62:0/2606783357 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f8e3c01a070 con 0x7f8e4c0a60b0 2024-04-19T01:46:44.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e3affd640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f8e2c036420 0x7f8e2c0388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:44.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e49d9b640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c13c780 0x7f8e4c13cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:44.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e3affd640 1 -- 172.21.15.62:0/2606783357 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(219..219 src has 108..219) ==== 4346+0+0 (secure 0 0 0) 0x7f8e3c049380 con 0x7f8e4c0a60b0 2024-04-19T01:46:44.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e49d9b640 10 monclient: get_auth_request con 0x7f8e4c13c780 auth_method 0 2024-04-19T01:46:44.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e49d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:44.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e49d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:44.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e49d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:44.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e49d9b640 1 -- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c13c780 msgr2=0x7f8e4c13cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-19T01:46:44.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e49d9b640 1 -- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c13c780 msgr2=0x7f8e4c13cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:46:44.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e49d9b640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c13c780 0x7f8e4c13cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:46:44.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e49d9b640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c13c780 0x7f8e4c13cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:46:44.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e4ad9d640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f8e2c036420 0x7f8e2c0388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:44.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.367+0000 7f8e4ad9d640 10 monclient: get_auth_request con 0x7f8e2c036420 auth_method 0 2024-04-19T01:46:44.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.371+0000 7f8e4ad9d640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f8e2c036420 0x7f8e2c0388e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8e44007fd0 tx=0x7f8e44009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:44.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.571+0000 7f8e49d9b640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c13c780 0x7f8e4c13cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:44.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.571+0000 7f8e49d9b640 10 monclient: get_auth_request con 0x7f8e4c13c780 auth_method 0 2024-04-19T01:46:44.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.571+0000 7f8e49d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:44.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.571+0000 7f8e49d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:44.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.571+0000 7f8e49d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:46:44.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.571+0000 7f8e49d9b640 1 -- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c13c780 msgr2=0x7f8e4c13cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-19T01:46:44.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.571+0000 7f8e49d9b640 1 -- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c13c780 msgr2=0x7f8e4c13cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:46:44.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.571+0000 7f8e49d9b640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c13c780 0x7f8e4c13cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:46:44.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.571+0000 7f8e49d9b640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c13c780 0x7f8e4c13cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:46:44.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.975+0000 7f8e49d9b640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c13c780 0x7f8e4c13cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:44.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.975+0000 7f8e49d9b640 10 monclient: get_auth_request con 0x7f8e4c13c780 auth_method 0 2024-04-19T01:46:44.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.975+0000 7f8e49d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:44.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.975+0000 7f8e49d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:44.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.975+0000 7f8e49d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:46:44.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.975+0000 7f8e49d9b640 1 -- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c13c780 msgr2=0x7f8e4c13cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-19T01:46:44.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.975+0000 7f8e49d9b640 1 -- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c13c780 msgr2=0x7f8e4c13cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:46:44.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.975+0000 7f8e49d9b640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c13c780 0x7f8e4c13cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:46:44.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:44.975+0000 7f8e49d9b640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c13c780 0x7f8e4c13cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:46:45.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:45.779+0000 7f8e49d9b640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c13c780 0x7f8e4c13cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:45.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:45.779+0000 7f8e49d9b640 10 monclient: get_auth_request con 0x7f8e4c13c780 auth_method 0 2024-04-19T01:46:45.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:45.779+0000 7f8e49d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:45.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:45.779+0000 7f8e49d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:45.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:45.779+0000 7f8e49d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:46:45.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:45.779+0000 7f8e49d9b640 1 -- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c13c780 msgr2=0x7f8e4c13cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-19T01:46:45.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:45.779+0000 7f8e49d9b640 1 -- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c13c780 msgr2=0x7f8e4c13cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:46:45.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:45.779+0000 7f8e49d9b640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c13c780 0x7f8e4c13cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:46:45.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:45.779+0000 7f8e49d9b640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c13c780 0x7f8e4c13cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:46:46.479 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:46.480 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30431]: starting. 2024-04-19T01:46:46.480 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30431]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-30396 2024-04-19T01:46:46.480 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30431]: created object 0... 2024-04-19T01:46:46.480 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30431]: created object 25... 2024-04-19T01:46:46.480 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30431]: created object 49... 2024-04-19T01:46:46.480 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30431]: finishing. 2024-04-19T01:46:46.480 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30431]: shutting down. 2024-04-19T01:46:46.540 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:46.540 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30433]: starting. 2024-04-19T01:46:46.540 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30433]: listing objects. 2024-04-19T01:46:46.540 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30433]: listed object 0... 2024-04-19T01:46:46.540 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30433]: listed object 25... 2024-04-19T01:46:46.541 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30433]: saw 50 objects 2024-04-19T01:46:46.541 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30433]: shutting down. 2024-04-19T01:46:47.266 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:47.266 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30432]: starting. 2024-04-19T01:46:47.266 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30432]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-30396 2024-04-19T01:46:47.266 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30432]: shutting down. 2024-04-19T01:46:47.269 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:47.269 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:46:47.269 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:46:47.272 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:46:47.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:47.363+0000 7f8e39ffb640 10 monclient: tick 2024-04-19T01:46:47.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:47.363+0000 7f8e39ffb640 10 monclient: _check_auth_tickets 2024-04-19T01:46:47.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:47.363+0000 7f8e39ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:46:47.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:47.363+0000 7f8e39ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:46:47.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:47.383+0000 7f8e49d9b640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c13c780 0x7f8e4c13cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:47.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:47.383+0000 7f8e49d9b640 10 monclient: get_auth_request con 0x7f8e4c13c780 auth_method 0 2024-04-19T01:46:47.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:47.383+0000 7f8e49d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:47.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:47.383+0000 7f8e49d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:47.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:47.383+0000 7f8e49d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:46:47.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:47.383+0000 7f8e49d9b640 1 -- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c13c780 msgr2=0x7f8e4c13cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-19T01:46:47.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:47.383+0000 7f8e49d9b640 1 -- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c13c780 msgr2=0x7f8e4c13cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:46:47.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:47.383+0000 7f8e49d9b640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c13c780 0x7f8e4c13cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:46:47.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:47.383+0000 7f8e49d9b640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c13c780 0x7f8e4c13cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-19T01:46:48.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.363+0000 7f8e39ffb640 10 monclient: tick 2024-04-19T01:46:48.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.363+0000 7f8e39ffb640 10 monclient: _check_auth_tickets 2024-04-19T01:46:48.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.363+0000 7f8e39ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:46:48.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.363+0000 7f8e39ffb640 5 monclient: _check_tell_commands timeout tell command 1 2024-04-19T01:46:48.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.363+0000 7f8e39ffb640 1 -- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c13c780 msgr2=0x7f8e4c13cb60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:46:48.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.363+0000 7f8e39ffb640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c13c780 0x7f8e4c13cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:48.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.363+0000 7f8e39ffb640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e24003550 0x7f8e240059e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:48.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.363+0000 7f8e39ffb640 1 -- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c13c780 msgr2=0x7f8e4c13cb60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:46:48.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.363+0000 7f8e39ffb640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c13c780 0x7f8e4c13cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:48.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.363+0000 7f8e39ffb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:48.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.363+0000 7f8e39ffb640 1 -- 172.21.15.62:0/2606783357 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4c120670 con 0x7f8e24003550 2024-04-19T01:46:48.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.367+0000 7f8e49d9b640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e24003550 0x7f8e240059e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:48.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.367+0000 7f8e49d9b640 10 monclient: get_auth_request con 0x7f8e24003550 auth_method 0 2024-04-19T01:46:48.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.367+0000 7f8e49d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:48.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.367+0000 7f8e49d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:48.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.367+0000 7f8e49d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:48.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.367+0000 7f8e49d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:48.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.367+0000 7f8e49d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:48.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.367+0000 7f8e49d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:48.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.367+0000 7f8e49d9b640 10 monclient(hunting): handle_auth_done global_id 24100 payload 824 2024-04-19T01:46:48.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.367+0000 7f8e49d9b640 1 -- 172.21.15.62:0/2606783357 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8e240060c0 con 0x7f8e24003550 2024-04-19T01:46:48.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.367+0000 7f8e49d9b640 1 -- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e24003550 msgr2=0x7f8e240059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:48.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.367+0000 7f8e49d9b640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e24003550 0x7f8e240059e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c09bc20 tx=0x7f8e40018f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:48.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.371+0000 7f8e3affd640 1 -- 172.21.15.62:0/2606783357 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e400c7070 con 0x7f8e24003550 2024-04-19T01:46:48.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.371+0000 7f8e3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:48.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.371+0000 7f8e3affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:46:48.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.371+0000 7f8e3affd640 10 monclient: dump: 2024-04-19T01:46:48.369 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:48.369 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:48.369 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:48.369 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:48.369 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:48.369 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:48.369 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:48.369 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:48.369 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:48.369 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:48.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.371+0000 7f8e3affd640 1 -- 172.21.15.62:0/2606783357 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8e400c0830 con 0x7f8e24003550 2024-04-19T01:46:48.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.371+0000 7f8e3affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:48.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.371+0000 7f8e3affd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:46:48.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.371+0000 7f8e3affd640 1 -- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e24003550 msgr2=0x7f8e240059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:48.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.371+0000 7f8e3affd640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e24003550 0x7f8e240059e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c09bc20 tx=0x7f8e40018f20 comp rx=0 tx=0).stop 2024-04-19T01:46:48.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.371+0000 7f8e3affd640 1 -- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e24003550 msgr2=0x7f8e240059e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:46:48.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.371+0000 7f8e3affd640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e24003550 0x7f8e240059e0 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c09bc20 tx=0x7f8e40018f20 comp rx=0 tx=0).stop 2024-04-19T01:46:48.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.379+0000 7f8e50be6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:46:48.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.379+0000 7f8e50be6640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c1384b0 0x7f8e4c1388c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:48.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.379+0000 7f8e50be6640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:48.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.379+0000 7f8e50be6640 1 -- 172.21.15.62:0/2606783357 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4c0a2b50 con 0x7f8e4c1384b0 2024-04-19T01:46:48.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.379+0000 7f8e49d9b640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c1384b0 0x7f8e4c1388c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:48.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.379+0000 7f8e49d9b640 10 monclient: get_auth_request con 0x7f8e4c1384b0 auth_method 0 2024-04-19T01:46:48.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.379+0000 7f8e49d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:48.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.379+0000 7f8e49d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:48.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.379+0000 7f8e49d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:48.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.379+0000 7f8e49d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:48.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.379+0000 7f8e49d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:48.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.379+0000 7f8e49d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:48.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.379+0000 7f8e49d9b640 10 monclient(hunting): handle_auth_done global_id 24103 payload 824 2024-04-19T01:46:48.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.379+0000 7f8e49d9b640 1 -- 172.21.15.62:0/2606783357 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8e4c068710 con 0x7f8e4c1384b0 2024-04-19T01:46:48.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.383+0000 7f8e49d9b640 1 -- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c1384b0 msgr2=0x7f8e4c1388c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:48.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.383+0000 7f8e49d9b640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c1384b0 0x7f8e4c1388c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8e4001ca20 tx=0x7f8e40009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:48.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.383+0000 7f8e3affd640 1 -- 172.21.15.62:0/2606783357 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e400c7050 con 0x7f8e4c1384b0 2024-04-19T01:46:48.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.383+0000 7f8e3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:48.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.383+0000 7f8e3affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:46:48.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.383+0000 7f8e3affd640 10 monclient: dump: 2024-04-19T01:46:48.381 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:48.381 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:48.381 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:48.381 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:48.382 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:48.382 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:48.382 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:48.382 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:48.382 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:48.382 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:48.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.383+0000 7f8e3affd640 1 -- 172.21.15.62:0/2606783357 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7f8e400c8040 con 0x7f8e4c1384b0 2024-04-19T01:46:48.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.383+0000 7f8e3affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:46:48.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.383+0000 7f8e3affd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:46:48.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.383+0000 7f8e3affd640 1 -- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c1384b0 msgr2=0x7f8e4c1388c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:48.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.383+0000 7f8e3affd640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c1384b0 0x7f8e4c1388c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8e4001ca20 tx=0x7f8e40009190 comp rx=0 tx=0).stop 2024-04-19T01:46:48.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.383+0000 7f8e3affd640 1 -- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c1384b0 msgr2=0x7f8e4c1388c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:46:48.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.383+0000 7f8e3affd640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c1384b0 0x7f8e4c1388c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:48.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.383+0000 7f8e4ad9d640 1 -- 172.21.15.62:0/2606783357 reap_dead start 2024-04-19T01:46:48.383 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-19T01:46:48.383 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "b", 2024-04-19T01:46:48.384 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 1, 2024-04-19T01:46:48.384 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "peon", 2024-04-19T01:46:48.384 INFO:teuthology.orchestra.run.smithi062.stdout: "election_epoch": 34, 2024-04-19T01:46:48.384 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 15031, 2024-04-19T01:46:48.384 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum": [ 2024-04-19T01:46:48.384 INFO:teuthology.orchestra.run.smithi062.stdout: 0, 2024-04-19T01:46:48.384 INFO:teuthology.orchestra.run.smithi062.stdout: 1, 2024-04-19T01:46:48.384 INFO:teuthology.orchestra.run.smithi062.stdout: 2 2024-04-19T01:46:48.384 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:46:48.384 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_age": 4, 2024-04-19T01:46:48.384 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:46:48.384 INFO:teuthology.orchestra.run.smithi062.stdout: "required_con": "2450521708007866372", 2024-04-19T01:46:48.384 INFO:teuthology.orchestra.run.smithi062.stdout: "required_mon": [ 2024-04-19T01:46:48.384 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:46:48.384 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:46:48.384 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:46:48.385 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:46:48.385 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:46:48.385 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:46:48.385 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:46:48.385 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:46:48.385 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:46:48.385 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:46:48.385 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:46:48.385 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:46:48.385 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:46:48.385 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_mon": [ 2024-04-19T01:46:48.385 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:46:48.385 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:46:48.385 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:46:48.385 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:46:48.385 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:46:48.385 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:46:48.386 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:46:48.386 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:46:48.386 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:46:48.386 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:46:48.386 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:46:48.386 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:48.386 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:48.386 INFO:teuthology.orchestra.run.smithi062.stdout: "outside_quorum": [], 2024-04-19T01:46:48.386 INFO:teuthology.orchestra.run.smithi062.stdout: "extra_probe_peers": [], 2024-04-19T01:46:48.386 INFO:teuthology.orchestra.run.smithi062.stdout: "sync_provider": [], 2024-04-19T01:46:48.386 INFO:teuthology.orchestra.run.smithi062.stdout: "monmap": { 2024-04-19T01:46:48.386 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 1, 2024-04-19T01:46:48.386 INFO:teuthology.orchestra.run.smithi062.stdout: "fsid": "6466e4b6-2520-485f-95b3-9401915fbbad", 2024-04-19T01:46:48.386 INFO:teuthology.orchestra.run.smithi062.stdout: "modified": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:46:48.386 INFO:teuthology.orchestra.run.smithi062.stdout: "created": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:46:48.386 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release": 19, 2024-04-19T01:46:48.387 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release_name": "squid", 2024-04-19T01:46:48.387 INFO:teuthology.orchestra.run.smithi062.stdout: "election_strategy": 1, 2024-04-19T01:46:48.387 INFO:teuthology.orchestra.run.smithi062.stdout: "disallowed_leaders: ": "", 2024-04-19T01:46:48.387 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false, 2024-04-19T01:46:48.387 INFO:teuthology.orchestra.run.smithi062.stdout: "tiebreaker_mon": "", 2024-04-19T01:46:48.387 INFO:teuthology.orchestra.run.smithi062.stdout: "removed_ranks: ": "", 2024-04-19T01:46:48.387 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:46:48.387 INFO:teuthology.orchestra.run.smithi062.stdout: "persistent": [ 2024-04-19T01:46:48.387 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:46:48.387 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:46:48.387 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:46:48.387 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:46:48.387 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:46:48.387 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:46:48.387 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:46:48.387 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:46:48.387 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:46:48.388 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:46:48.388 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:46:48.388 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:46:48.388 INFO:teuthology.orchestra.run.smithi062.stdout: "optional": [] 2024-04-19T01:46:48.388 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:48.388 INFO:teuthology.orchestra.run.smithi062.stdout: "mons": [ 2024-04-19T01:46:48.388 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:48.388 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 0, 2024-04-19T01:46:48.388 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "a", 2024-04-19T01:46:48.388 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:46:48.388 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:46:48.388 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:48.388 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:46:48.388 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3300", 2024-04-19T01:46:48.388 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:48.388 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:48.388 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:48.389 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:46:48.389 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789", 2024-04-19T01:46:48.389 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:48.389 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:48.389 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:48.389 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:48.389 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789/0", 2024-04-19T01:46:48.389 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6789/0", 2024-04-19T01:46:48.389 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:46:48.389 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:46:48.389 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:46:48.389 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:48.389 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:48.389 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 1, 2024-04-19T01:46:48.389 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "b", 2024-04-19T01:46:48.389 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:46:48.389 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:46:48.390 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:48.390 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:46:48.390 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:3300", 2024-04-19T01:46:48.390 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:48.390 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:48.390 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:48.390 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:46:48.390 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789", 2024-04-19T01:46:48.390 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:48.390 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:48.390 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:48.390 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:48.390 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789/0", 2024-04-19T01:46:48.390 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.71:6789/0", 2024-04-19T01:46:48.390 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:46:48.391 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:46:48.391 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:46:48.391 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:48.391 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:48.391 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 2, 2024-04-19T01:46:48.391 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "c", 2024-04-19T01:46:48.391 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:46:48.391 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:46:48.391 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:48.391 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:46:48.391 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3301", 2024-04-19T01:46:48.391 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:48.391 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:48.391 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:48.391 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:46:48.391 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790", 2024-04-19T01:46:48.392 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:48.392 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:48.392 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:48.392 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:48.392 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790/0", 2024-04-19T01:46:48.392 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6790/0", 2024-04-19T01:46:48.392 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:46:48.392 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:46:48.392 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:46:48.392 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:48.392 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:48.392 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:48.392 INFO:teuthology.orchestra.run.smithi062.stdout: "feature_map": { 2024-04-19T01:46:48.392 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": [ 2024-04-19T01:46:48.392 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:48.392 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:46:48.393 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:46:48.393 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:46:48.393 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:48.393 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:46:48.393 INFO:teuthology.orchestra.run.smithi062.stdout: "client": [ 2024-04-19T01:46:48.393 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:48.393 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:46:48.393 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:46:48.393 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:46:48.393 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:48.393 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:48.393 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:48.393 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false 2024-04-19T01:46:48.393 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-19T01:46:48.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.383+0000 7f8e50be6640 1 -- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f8e2c036420 msgr2=0x7f8e2c0388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:48.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.383+0000 7f8e50be6640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f8e2c036420 0x7f8e2c0388e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8e44007fd0 tx=0x7f8e44009040 comp rx=0 tx=0).stop 2024-04-19T01:46:48.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.383+0000 7f8e50be6640 10 monclient: shutdown 2024-04-19T01:46:48.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.383+0000 7f8e50be6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:46:48.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.383+0000 7f8e50be6640 1 -- 172.21.15.62:0/2606783357 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8e4c0a60b0 msgr2=0x7f8e4c1cf5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:48.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.383+0000 7f8e50be6640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8e4c0a60b0 0x7f8e4c1cf5e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c00ea30 tx=0x7f8e3c00eef0 comp rx=0 tx=0).stop 2024-04-19T01:46:48.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.383+0000 7f8e50be6640 1 -- 172.21.15.62:0/2606783357 shutdown_connections 2024-04-19T01:46:48.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.383+0000 7f8e50be6640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f8e2c036420 0x7f8e2c0388e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:48.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.383+0000 7f8e50be6640 1 --2- 172.21.15.62:0/2606783357 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f8e4c0a60b0 0x7f8e4c1cf5e0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:48.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.383+0000 7f8e50be6640 1 -- 172.21.15.62:0/2606783357 >> 172.21.15.62:0/2606783357 conn(0x7f8e4c0aa1d0 msgr2=0x7f8e4c0a79d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:46:48.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.383+0000 7f8e50be6640 1 -- 172.21.15.62:0/2606783357 shutdown_connections 2024-04-19T01:46:48.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.383+0000 7f8e50be6640 1 -- 172.21.15.62:0/2606783357 wait complete. 2024-04-19T01:46:48.410 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-19T01:46:48.410 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-19T01:46:48.410 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-19T01:46:48.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.551+0000 7fba0c7e2640 10 monclient: get_monmap_and_config 2024-04-19T01:46:48.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.551+0000 7fba0c7e2640 10 monclient: build_initial_monmap 2024-04-19T01:46:48.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.551+0000 7fba0c7e2640 10 monclient: monmap: 2024-04-19T01:46:48.550 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:46:48.550 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:48.550 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:46:48.556110+0000 2024-04-19T01:46:48.550 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:46:48.556110+0000 2024-04-19T01:46:48.550 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:46:48.551 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:48.551 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:46:48.551 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:46:48.551 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:46:48.551 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:46:48.551 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:46:48.551 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:48.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.551+0000 7fba0c7e2640 1 Processor -- start 2024-04-19T01:46:48.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.551+0000 7fba0c7e2640 1 -- start start 2024-04-19T01:46:48.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.551+0000 7fba0c7e2640 10 monclient: init 2024-04-19T01:46:48.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.551+0000 7fba0c7e2640 10 monclient: _reopen_session rank -1 2024-04-19T01:46:48.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.551+0000 7fba0c7e2640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-04-19T01:46:48.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.551+0000 7fba0c7e2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba040a47e0 0x7fba040a4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:48.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.551+0000 7fba0c7e2640 10 monclient(hunting): picked mon.noname-b con 0x7fba040a47e0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:46:48.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.551+0000 7fba0c7e2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fba040a5100 addr v1:172.21.15.62:6789/0 2024-04-19T01:46:48.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.551+0000 7fba0c7e2640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fba041357c0 0x7fba0413d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:48.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.551+0000 7fba0c7e2640 10 monclient(hunting): picked mon.noname-c con 0x7fba041357c0 addr v2:172.21.15.71:3300/0 2024-04-19T01:46:48.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.551+0000 7fba0c7e2640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba0409f7c0 con 0x7fba040a5100 2024-04-19T01:46:48.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.551+0000 7fba0c7e2640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:48.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.551+0000 7fba0c7e2640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fba0409bf00 con 0x7fba041357c0 2024-04-19T01:46:48.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.551+0000 7fba0c7e2640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:48.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.551+0000 7fba0c7e2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fba0409ee90 con 0x7fba040a47e0 2024-04-19T01:46:48.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.551+0000 7fba0c7e2640 10 monclient(hunting): _renew_subs 2024-04-19T01:46:48.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.551+0000 7fba0c7e2640 10 monclient(hunting): authenticate will time out at 1194.375560s 2024-04-19T01:46:48.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.551+0000 7fba09d7d640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fba040a5100 0x7fba041350b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:39800/0 (socket says 172.21.15.62:39800) 2024-04-19T01:46:48.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.551+0000 7fba09d7d640 1 -- 172.21.15.62:0/99647857 learned_addr learned my addr 172.21.15.62:0/99647857 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:46:48.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.551+0000 7fba0a57e640 1 --2- 172.21.15.62:0/99647857 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba040a47e0 0x7fba040a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:48.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.551+0000 7fba0a57e640 10 monclient(hunting): get_auth_request con 0x7fba040a47e0 auth_method 0 2024-04-19T01:46:48.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.551+0000 7fba0a57e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:48.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.551+0000 7fba0a57e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:48.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.551+0000 7fba0a57e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:48.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.551+0000 7fba0a57e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:48.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.551+0000 7fba0a57e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:48.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.555+0000 7fba0a57e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:48.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.555+0000 7fba0ad7f640 1 --2- 172.21.15.62:0/99647857 >> v2:172.21.15.71:3300/0 conn(0x7fba041357c0 0x7fba0413d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:48.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.555+0000 7fba0ad7f640 10 monclient(hunting): get_auth_request con 0x7fba041357c0 auth_method 0 2024-04-19T01:46:48.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.555+0000 7fba0ad7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:48.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.555+0000 7fba0ad7f640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:48.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.555+0000 7fba0ad7f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:48.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.555+0000 7fba0a57e640 10 monclient(hunting): handle_auth_done global_id 5300 payload 274 2024-04-19T01:46:48.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.555+0000 7fba0a57e640 1 -- 172.21.15.62:0/99647857 >> v2:172.21.15.71:3300/0 conn(0x7fba041357c0 msgr2=0x7fba0413d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:46:48.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.555+0000 7fba0a57e640 1 --2- 172.21.15.62:0/99647857 >> v2:172.21.15.71:3300/0 conn(0x7fba041357c0 0x7fba0413d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:48.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.555+0000 7fba0a57e640 1 -- 172.21.15.62:0/99647857 >> v1:172.21.15.62:6789/0 conn(0x7fba040a5100 legacy=0x7fba041350b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:46:48.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.555+0000 7fba0a57e640 10 monclient: _finish_hunting 0 2024-04-19T01:46:48.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.555+0000 7fba0a57e640 1 monclient: found mon.noname-b 2024-04-19T01:46:48.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.555+0000 7fba0a57e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:46:48.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.555+0000 7fba0a57e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:46:48.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.555+0000 7fba0a57e640 1 -- 172.21.15.62:0/99647857 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba0409c2d0 con 0x7fba040a47e0 2024-04-19T01:46:48.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.559+0000 7fba0a57e640 1 -- 172.21.15.62:0/99647857 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba040a47e0 msgr2=0x7fba040a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:48.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.559+0000 7fba0a57e640 1 --2- 172.21.15.62:0/99647857 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba040a47e0 0x7fba040a4bc0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb9f0009d90 tx=0x7fb9f001c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb143f51f04410ca server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:48.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.559+0000 7fba0957c640 1 -- 172.21.15.62:0/99647857 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f00ac070 con 0x7fba040a47e0 2024-04-19T01:46:48.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.559+0000 7fba0957c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:48.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.559+0000 7fba0957c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:46:48.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.559+0000 7fba0957c640 10 monclient: dump: 2024-04-19T01:46:48.558 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:48.558 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:48.558 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:48.559 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:48.559 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:48.559 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:48.559 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:48.559 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:48.559 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:48.559 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:48.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.559+0000 7fba0957c640 10 monclient: _finish_auth 0 2024-04-19T01:46:48.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.559+0000 7fba0957c640 10 monclient: _check_auth_tickets 2024-04-19T01:46:48.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.559+0000 7fba0957c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:46:48.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.559+0000 7fba0957c640 1 -- 172.21.15.62:0/99647857 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9f00a6640 con 0x7fba040a47e0 2024-04-19T01:46:48.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.559+0000 7fba0c7e2640 5 monclient: authenticate success, global_id 5300 2024-04-19T01:46:48.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.559+0000 7fba0c7e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:46:48.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.559+0000 7fba0957c640 10 monclient: handle_config config(0 keys) 2024-04-19T01:46:48.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.559+0000 7fba0c7e2640 10 monclient: get_monmap_and_config success 2024-04-19T01:46:48.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.559+0000 7fba0957c640 1 -- 172.21.15.62:0/99647857 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f00a6a10 con 0x7fba040a47e0 2024-04-19T01:46:48.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.559+0000 7fba0957c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:48.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.559+0000 7fba0c7e2640 4 set_mon_vals no callback set 2024-04-19T01:46:48.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.559+0000 7fba0957c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:46:48.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.559+0000 7fba0957c640 10 monclient: dump: 2024-04-19T01:46:48.560 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:48.560 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:48.560 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:48.560 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:48.560 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:48.560 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:48.560 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:48.560 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:48.560 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:48.561 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:48.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.559+0000 7fba0c7e2640 10 monclient: shutdown 2024-04-19T01:46:48.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.559+0000 7fba0c7e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:46:48.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.559+0000 7fba0c7e2640 1 -- 172.21.15.62:0/99647857 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba040a47e0 msgr2=0x7fba040a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:48.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.559+0000 7fba0c7e2640 1 --2- 172.21.15.62:0/99647857 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba040a47e0 0x7fba040a4bc0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb9f0009d90 tx=0x7fb9f001c6e0 comp rx=0 tx=0).stop 2024-04-19T01:46:48.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.559+0000 7fba0c7e2640 1 -- 172.21.15.62:0/99647857 shutdown_connections 2024-04-19T01:46:48.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.559+0000 7fba0c7e2640 1 --2- 172.21.15.62:0/99647857 >> v2:172.21.15.71:3300/0 conn(0x7fba041357c0 0x7fba0413d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:48.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.559+0000 7fba0c7e2640 1 --2- 172.21.15.62:0/99647857 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba040a47e0 0x7fba040a4bc0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:48.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.559+0000 7fba0c7e2640 1 -- 172.21.15.62:0/99647857 >> 172.21.15.62:0/99647857 conn(0x7fba040a23d0 msgr2=0x7fba040a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:46:48.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.559+0000 7fba0c7e2640 1 -- 172.21.15.62:0/99647857 shutdown_connections 2024-04-19T01:46:48.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.559+0000 7fba0c7e2640 1 -- 172.21.15.62:0/99647857 wait complete. 2024-04-19T01:46:48.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.559+0000 7fba0c7e2640 10 monclient: build_initial_monmap 2024-04-19T01:46:48.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.559+0000 7fba0c7e2640 10 monclient: monmap: 2024-04-19T01:46:48.566 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:46:48.566 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:46:48.566 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:46:48.566 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:46:48.566 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:46:48.566 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:48.566 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:46:48.567 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:46:48.567 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:46:48.567 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:48.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0c7e2640 1 Processor -- start 2024-04-19T01:46:48.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0c7e2640 1 -- start start 2024-04-19T01:46:48.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0c7e2640 10 monclient: init 2024-04-19T01:46:48.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0c7e2640 10 monclient: _reopen_session rank -1 2024-04-19T01:46:48.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0c7e2640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-19T01:46:48.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0c7e2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fba040a47e0 0x7fba0413c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:48.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0c7e2640 10 monclient(hunting): picked mon.noname-b con 0x7fba040a47e0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:46:48.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0c7e2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba040a5100 0x7fba0412e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:48.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0c7e2640 10 monclient(hunting): picked mon.noname-a con 0x7fba040a5100 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:46:48.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0c7e2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba041357c0 0x7fba0412e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:48.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0c7e2640 10 monclient(hunting): picked mon.noname-c con 0x7fba041357c0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:46:48.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0c7e2640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:48.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0c7e2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fba0409c8e0 con 0x7fba040a5100 2024-04-19T01:46:48.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0c7e2640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:48.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0c7e2640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fba0409c760 con 0x7fba040a47e0 2024-04-19T01:46:48.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0c7e2640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:48.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0c7e2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fba04140360 con 0x7fba041357c0 2024-04-19T01:46:48.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0c7e2640 10 monclient(hunting): _renew_subs 2024-04-19T01:46:48.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0c7e2640 10 monclient(hunting): authenticate will time out at 1194.384826s 2024-04-19T01:46:48.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0ad7f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba041357c0 0x7fba0412e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:48.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0ad7f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba041357c0 0x7fba0412e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:53546/0 (socket says 172.21.15.62:53546) 2024-04-19T01:46:48.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0ad7f640 1 -- 172.21.15.62:0/2773108922 learned_addr learned my addr 172.21.15.62:0/2773108922 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:46:48.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0ad7f640 10 monclient(hunting): get_auth_request con 0x7fba041357c0 auth_method 0 2024-04-19T01:46:48.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0ad7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:48.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0ad7f640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:48.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0ad7f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:48.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba09d7d640 1 --2- 172.21.15.62:0/2773108922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba040a5100 0x7fba0412e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:48.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba09d7d640 10 monclient(hunting): get_auth_request con 0x7fba040a5100 auth_method 0 2024-04-19T01:46:48.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba09d7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:48.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba09d7d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:48.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba09d7d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:48.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0ad7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:48.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0ad7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:48.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0ad7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:48.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba09d7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:48.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba09d7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:48.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba09d7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:48.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0a57e640 1 --2- 172.21.15.62:0/2773108922 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fba040a47e0 0x7fba0413c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:48.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0a57e640 10 monclient(hunting): get_auth_request con 0x7fba040a47e0 auth_method 0 2024-04-19T01:46:48.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0a57e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:48.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0a57e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:48.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0a57e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:48.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0ad7f640 10 monclient(hunting): handle_auth_done global_id 5303 payload 824 2024-04-19T01:46:48.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0ad7f640 1 -- 172.21.15.62:0/2773108922 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fba040a47e0 msgr2=0x7fba0413c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:48.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0ad7f640 1 --2- 172.21.15.62:0/2773108922 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fba040a47e0 0x7fba0413c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:48.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0ad7f640 1 -- 172.21.15.62:0/2773108922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba040a5100 msgr2=0x7fba0412e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:48.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0ad7f640 1 --2- 172.21.15.62:0/2773108922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba040a5100 0x7fba0412e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:48.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0ad7f640 10 monclient: _finish_hunting 0 2024-04-19T01:46:48.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0ad7f640 1 monclient: found mon.noname-c 2024-04-19T01:46:48.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0ad7f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:46:48.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0ad7f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:46:48.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0ad7f640 1 -- 172.21.15.62:0/2773108922 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba0412ee80 con 0x7fba041357c0 2024-04-19T01:46:48.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0ad7f640 1 -- 172.21.15.62:0/2773108922 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba041357c0 msgr2=0x7fba0412e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:48.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba0ad7f640 1 --2- 172.21.15.62:0/2773108922 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba041357c0 0x7fba0412e940 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc00d830 tx=0x7fb9fc00dcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:48.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba02ffd640 1 -- 172.21.15.62:0/2773108922 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc00b8d0 con 0x7fba041357c0 2024-04-19T01:46:48.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:48.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba02ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:46:48.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba02ffd640 10 monclient: dump: 2024-04-19T01:46:48.571 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:48.571 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:48.571 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:48.571 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:48.571 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:48.571 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:48.571 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:48.571 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:48.571 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:48.571 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:48.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba02ffd640 10 monclient: _finish_auth 0 2024-04-19T01:46:48.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba02ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:46:48.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:46:48.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba02ffd640 1 -- 172.21.15.62:0/2773108922 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9fc019070 con 0x7fba041357c0 2024-04-19T01:46:48.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba02ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:46:48.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba02ffd640 1 -- 172.21.15.62:0/2773108922 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc00bcb0 con 0x7fba041357c0 2024-04-19T01:46:48.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:48.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba02ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:46:48.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba02ffd640 10 monclient: dump: 2024-04-19T01:46:48.572 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:48.572 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:48.572 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:48.572 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:48.572 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:48.572 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:48.572 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:48.572 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:48.572 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:48.572 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:48.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.563+0000 7fba08d7b640 4 set_mon_vals no callback set 2024-04-19T01:46:48.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.567+0000 7fba0c7e2640 5 monclient: authenticate success, global_id 5303 2024-04-19T01:46:48.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.567+0000 7fba0c7e2640 10 monclient: _renew_subs 2024-04-19T01:46:48.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.567+0000 7fba0c7e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:46:48.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.567+0000 7fba0c7e2640 1 -- 172.21.15.62:0/2773108922 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba0413a640 con 0x7fba041357c0 2024-04-19T01:46:48.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.567+0000 7fba0c7e2640 10 monclient: _renew_subs 2024-04-19T01:46:48.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.567+0000 7fba0c7e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:46:48.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.567+0000 7fba0c7e2640 1 -- 172.21.15.62:0/2773108922 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fba0413b750 con 0x7fba041357c0 2024-04-19T01:46:48.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.567+0000 7fba0c7e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:48.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.567+0000 7fba0c7e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:48.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.567+0000 7fba0c7e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:46:48.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.567+0000 7fba0c7e2640 1 -- 172.21.15.62:0/2773108922 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba040692b0 con 0x7fba041357c0 2024-04-19T01:46:48.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.567+0000 7fba02ffd640 1 -- 172.21.15.62:0/2773108922 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fb9fc01e460 con 0x7fba041357c0 2024-04-19T01:46:48.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.567+0000 7fba02ffd640 1 --2- 172.21.15.62:0/2773108922 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fb9e80364a0 0x7fb9e8038960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:48.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.567+0000 7fba02ffd640 1 -- 172.21.15.62:0/2773108922 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(223..223 src has 108..223) ==== 4738+0+0 (secure 0 0 0) 0x7fb9fc0498e0 con 0x7fba041357c0 2024-04-19T01:46:48.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.567+0000 7fba0a57e640 1 --2- 172.21.15.62:0/2773108922 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fb9e80364a0 0x7fb9e8038960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:48.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.567+0000 7fba0a57e640 10 monclient: get_auth_request con 0x7fb9e80364a0 auth_method 0 2024-04-19T01:46:48.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.571+0000 7fba0a57e640 1 --2- 172.21.15.62:0/2773108922 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fb9e80364a0 0x7fb9e8038960 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fba0406cc30 tx=0x7fb9f00a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:48.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.571+0000 7fba02ffd640 1 -- 172.21.15.62:0/2773108922 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fba0413a640 con 0x7fba041357c0 2024-04-19T01:46:48.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.571+0000 7fba02ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:48.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.571+0000 7fba02ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:46:48.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.799+0000 7fba0c7e2640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-19T01:46:48.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.799+0000 7fba0c7e2640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-19T01:46:48.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.799+0000 7fba0c7e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:46:48.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.799+0000 7fba0c7e2640 1 -- 172.21.15.62:0/2773108922 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fba0406b980 con 0x7fba041357c0 2024-04-19T01:46:48.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.803+0000 7fba02ffd640 1 -- 172.21.15.62:0/2773108922 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fb9fc00be50 con 0x7fba041357c0 2024-04-19T01:46:48.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.803+0000 7fba02ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-19T01:46:48.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.803+0000 7fba02ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:46:48.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.803+0000 7fba0c7e2640 1 -- 172.21.15.62:0/2773108922 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fb9e80364a0 msgr2=0x7fb9e8038960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:48.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.803+0000 7fba0c7e2640 1 --2- 172.21.15.62:0/2773108922 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fb9e80364a0 0x7fb9e8038960 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fba0406cc30 tx=0x7fb9f00a7000 comp rx=0 tx=0).stop 2024-04-19T01:46:48.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.803+0000 7fba0c7e2640 10 monclient: shutdown 2024-04-19T01:46:48.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.803+0000 7fba0c7e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:46:48.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.803+0000 7fba0c7e2640 1 -- 172.21.15.62:0/2773108922 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba041357c0 msgr2=0x7fba0412e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:48.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.803+0000 7fba0c7e2640 1 --2- 172.21.15.62:0/2773108922 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba041357c0 0x7fba0412e940 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc00d830 tx=0x7fb9fc00dcf0 comp rx=0 tx=0).stop 2024-04-19T01:46:48.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.803+0000 7fba0c7e2640 1 -- 172.21.15.62:0/2773108922 shutdown_connections 2024-04-19T01:46:48.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.803+0000 7fba0c7e2640 1 --2- 172.21.15.62:0/2773108922 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fb9e80364a0 0x7fb9e8038960 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:48.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.803+0000 7fba0c7e2640 1 --2- 172.21.15.62:0/2773108922 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fba041357c0 0x7fba0412e940 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:48.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.803+0000 7fba0c7e2640 1 --2- 172.21.15.62:0/2773108922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba040a5100 0x7fba0412e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:48.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.803+0000 7fba0c7e2640 1 --2- 172.21.15.62:0/2773108922 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fba040a47e0 0x7fba0413c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:48.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.803+0000 7fba0c7e2640 1 -- 172.21.15.62:0/2773108922 >> 172.21.15.62:0/2773108922 conn(0x7fba040a23d0 msgr2=0x7fba0406fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:46:48.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.803+0000 7fba0c7e2640 1 -- 172.21.15.62:0/2773108922 shutdown_connections 2024-04-19T01:46:48.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:48.807+0000 7fba0c7e2640 1 -- 172.21.15.62:0/2773108922 wait complete. 2024-04-19T01:46:48.809 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.811+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:46:48.810 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.811+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=8,config=1,health=12,kv=11,logm=68} crc {auth=1339670619,config=1228700967,health=3116775328,kv=3416605158,logm=1337300943}) 2024-04-19T01:46:48.810 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.811+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=9,config=2,health=12,kv=11,logm=66} crc {auth=3304502536,config=3541143443,health=3988873958,kv=3416605158,logm=1086499585}) 2024-04-19T01:46:48.810 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.811+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:46:48.810 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.811+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=8,config=1,health=12,kv=11,logm=68} crc {auth=1339670619,config=1228700967,health=3116775328,kv=3416605158,logm=1337300943}) 2024-04-19T01:46:48.810 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.811+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=7,config=2,health=12,kv=11,logm=68} crc {auth=3204345789,config=3541143443,health=3116775328,kv=3416605158,logm=3292718689}) 2024-04-19T01:46:48.812 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.811+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:46:48.812 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.811+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3806954737}) 2024-04-19T01:46:48.812 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.811+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3334336856}) 2024-04-19T01:46:48.812 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.811+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:46:48.812 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.811+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3806954737}) 2024-04-19T01:46:48.812 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.811+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1493112706}) 2024-04-19T01:46:48.813 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.815+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:46:48.814 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.815+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1739313530}) 2024-04-19T01:46:48.814 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.815+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3445344631}) 2024-04-19T01:46:48.814 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.815+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:46:48.814 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.815+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1739313530}) 2024-04-19T01:46:48.814 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.815+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=489566348}) 2024-04-19T01:46:48.815 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.815+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:46:48.815 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.815+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=141120486}) 2024-04-19T01:46:48.816 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.815+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2590735570}) 2024-04-19T01:46:48.816 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.815+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:46:48.816 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.815+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=141120486}) 2024-04-19T01:46:48.816 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.815+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2716892516}) 2024-04-19T01:46:48.817 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.819+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:46:48.817 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.819+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1482176158}) 2024-04-19T01:46:48.817 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.819+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3864532045}) 2024-04-19T01:46:48.818 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.819+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:46:48.818 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.819+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1482176158}) 2024-04-19T01:46:48.818 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.819+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3959398236}) 2024-04-19T01:46:48.819 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.819+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:46:48.819 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.819+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2751069269}) 2024-04-19T01:46:48.819 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.819+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3195470274}) 2024-04-19T01:46:48.819 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.819+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:46:48.820 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.819+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2751069269}) 2024-04-19T01:46:48.820 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.819+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3064000999}) 2024-04-19T01:46:48.821 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.823+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:46:48.821 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.823+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3766702990}) 2024-04-19T01:46:48.821 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.823+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3322683045}) 2024-04-19T01:46:48.821 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.823+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:46:48.821 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.823+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3766702990}) 2024-04-19T01:46:48.821 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.823+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2654026438}) 2024-04-19T01:46:48.823 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-19T01:46:48.823 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.823+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:46:48.823 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.823+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2010193864}) 2024-04-19T01:46:48.824 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.823+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1800093631}) 2024-04-19T01:46:48.824 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.823+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:46:48.824 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.823+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2010193864}) 2024-04-19T01:46:48.824 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.823+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3815979759}) 2024-04-19T01:46:48.825 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.827+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:46:48.825 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.827+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2864423228}) 2024-04-19T01:46:48.825 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.827+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3589005307}) 2024-04-19T01:46:48.825 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.827+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:46:48.825 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.827+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2864423228}) 2024-04-19T01:46:48.825 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.827+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3228415601}) 2024-04-19T01:46:48.827 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.827+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:46:48.827 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.827+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=709024105}) 2024-04-19T01:46:48.827 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.827+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3537340820}) 2024-04-19T01:46:48.827 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.827+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:46:48.827 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.827+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=709024105}) 2024-04-19T01:46:48.827 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.827+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=399372822}) 2024-04-19T01:46:48.830 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.831+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:46:48.830 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.831+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=49,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=5} crc {logm=4186217714,mdsmap=2840952586,mgr=3717952716,mgr_command_descs=3636332126,mgr_metadata=230233602,mgrstat=286149617,mon_config_key=18691498,monmap=1260512061,osd_metadata=2171542279,osd_pg_creating=2195851173,osdmap=713002542}) 2024-04-19T01:46:48.830 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.831+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=51,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=12,monmap=2,osd_metadata=4,osd_pg_creating=1,osdmap=9} crc {logm=3263108967,mdsmap=2840952586,mgr=3616979525,mgr_command_descs=3636332126,mgr_metadata=230233602,mgrstat=286149617,mon_config_key=2148922962,monmap=3140514009,osd_metadata=2003278329,osd_pg_creating=2195851173,osdmap=799314618}) 2024-04-19T01:46:48.830 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.831+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:46:48.830 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.831+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=49,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=5} crc {logm=4186217714,mdsmap=2840952586,mgr=3717952716,mgr_command_descs=3636332126,mgr_metadata=230233602,mgrstat=286149617,mon_config_key=18691498,monmap=1260512061,osd_metadata=2171542279,osd_pg_creating=2195851173,osdmap=713002542}) 2024-04-19T01:46:48.830 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.831+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=50,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=7} crc {logm=1179903947,mdsmap=2840952586,mgr=1735148285,mgr_command_descs=3636332126,mgr_metadata=230233602,mgrstat=2073167318,mon_config_key=18691498,monmap=1260512061,osd_metadata=2171542278,osd_pg_creating=2195851174,osdmap=1349458232}) 2024-04-19T01:46:48.832 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.831+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:46:48.832 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.831+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=34886405}) 2024-04-19T01:46:48.832 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.831+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3588192295}) 2024-04-19T01:46:48.832 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.831+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:46:48.832 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.831+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=34886405}) 2024-04-19T01:46:48.832 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.831+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2374184543}) 2024-04-19T01:46:48.840 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.843+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:46:48.840 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.843+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=81} crc {osdmap=290056678}) 2024-04-19T01:46:48.840 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.843+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=82} crc {osdmap=2880455419}) 2024-04-19T01:46:48.840 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.843+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:46:48.840 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.843+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=81} crc {osdmap=290056678}) 2024-04-19T01:46:48.841 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:46:48.843+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=75} crc {osdmap=2026982260}) 2024-04-19T01:46:49.722 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30485]: starting. 2024-04-19T01:46:49.722 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30485]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-30484 2024-04-19T01:46:49.722 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30485]: created object 0... 2024-04-19T01:46:49.723 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30485]: created object 25... 2024-04-19T01:46:49.723 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30485]: created object 49... 2024-04-19T01:46:49.723 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30485]: finishing. 2024-04-19T01:46:49.723 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30485]: shutting down. 2024-04-19T01:46:49.824 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-19T01:46:49.824 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:46:49.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbda206640 10 monclient: get_monmap_and_config 2024-04-19T01:46:49.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbda206640 10 monclient: build_initial_monmap 2024-04-19T01:46:49.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbda206640 10 monclient: monmap: 2024-04-19T01:46:49.954 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:46:49.954 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:49.954 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:46:49.959868+0000 2024-04-19T01:46:49.954 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:46:49.959868+0000 2024-04-19T01:46:49.955 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:46:49.955 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:49.955 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:46:49.955 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:46:49.955 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:46:49.955 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:46:49.955 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:46:49.955 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:49.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbda206640 1 Processor -- start 2024-04-19T01:46:49.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbda206640 1 -- start start 2024-04-19T01:46:49.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbda206640 10 monclient: init 2024-04-19T01:46:49.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbda206640 10 monclient: _reopen_session rank -1 2024-04-19T01:46:49.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbda206640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-04-19T01:46:49.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbda206640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcbd4137780 0x7fcbd4137b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:49.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbda206640 10 monclient(hunting): picked mon.noname-a con 0x7fcbd4137780 addr v2:172.21.15.62:3300/0 2024-04-19T01:46:49.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbda206640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcbd41380a0 addr v1:172.21.15.62:6789/0 2024-04-19T01:46:49.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbda206640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcbd40a60b0 0x7fcbd40a4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:49.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbda206640 10 monclient(hunting): picked mon.noname-b con 0x7fcbd40a60b0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:46:49.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbda206640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbd409c2e0 con 0x7fcbd41380a0 2024-04-19T01:46:49.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbda206640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:49.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbda206640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fcbd409bd20 con 0x7fcbd4137780 2024-04-19T01:46:49.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbda206640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:49.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbda206640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fcbd409c000 con 0x7fcbd40a60b0 2024-04-19T01:46:49.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbda206640 10 monclient(hunting): _renew_subs 2024-04-19T01:46:49.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbda206640 10 monclient(hunting): authenticate will time out at 1195.779551s 2024-04-19T01:46:49.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbd3fff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcbd40a60b0 0x7fcbd40a4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:49.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbd37fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcbd4137780 0x7fcbd4137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:49.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbd2ffd640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fcbd41380a0 0x7fcbd40a45d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:39812/0 (socket says 172.21.15.62:39812) 2024-04-19T01:46:49.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbd2ffd640 1 -- 172.21.15.62:0/538233784 learned_addr learned my addr 172.21.15.62:0/538233784 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:46:49.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbd37fe640 10 monclient(hunting): get_auth_request con 0x7fcbd4137780 auth_method 0 2024-04-19T01:46:49.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:49.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbd37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:49.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:49.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbd3fff640 10 monclient(hunting): get_auth_request con 0x7fcbd40a60b0 auth_method 0 2024-04-19T01:46:49.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:49.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbd3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:49.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:49.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbd37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:49.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbd37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:49.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbd37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:49.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbd3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:49.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbd3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:49.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.955+0000 7fcbd3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:49.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.959+0000 7fcbd3fff640 10 monclient(hunting): handle_auth_done global_id 5306 payload 274 2024-04-19T01:46:49.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.959+0000 7fcbd3fff640 1 -- 172.21.15.62:0/538233784 >> v2:172.21.15.62:3300/0 conn(0x7fcbd4137780 msgr2=0x7fcbd4137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:46:49.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.959+0000 7fcbd3fff640 1 --2- 172.21.15.62:0/538233784 >> v2:172.21.15.62:3300/0 conn(0x7fcbd4137780 0x7fcbd4137b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:49.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.959+0000 7fcbd3fff640 1 -- 172.21.15.62:0/538233784 >> v1:172.21.15.62:6789/0 conn(0x7fcbd41380a0 legacy=0x7fcbd40a45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:46:49.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.959+0000 7fcbd3fff640 10 monclient: _finish_hunting 0 2024-04-19T01:46:49.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.959+0000 7fcbd3fff640 1 monclient: found mon.noname-b 2024-04-19T01:46:49.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.959+0000 7fcbd3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:46:49.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.959+0000 7fcbd3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:46:49.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.959+0000 7fcbd3fff640 1 -- 172.21.15.62:0/538233784 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbd4073210 con 0x7fcbd40a60b0 2024-04-19T01:46:49.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.959+0000 7fcbd37fe640 1 --2- 172.21.15.62:0/538233784 >> v2:172.21.15.62:3300/0 conn(0x7fcbd4137780 0x7fcbd4137b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:46:49.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd3fff640 1 -- 172.21.15.62:0/538233784 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcbd40a60b0 msgr2=0x7fcbd40a4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:49.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd3fff640 1 --2- 172.21.15.62:0/538233784 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcbd40a60b0 0x7fcbd40a4ce0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc0097c0 tx=0x7fcbcc01c180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5dd74d0c4ce33b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:49.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd27fc640 1 -- 172.21.15.62:0/538233784 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc0b0070 con 0x7fcbd40a60b0 2024-04-19T01:46:49.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:49.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:46:49.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd27fc640 10 monclient: dump: 2024-04-19T01:46:49.962 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:49.962 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:49.962 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:49.962 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:49.962 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:49.962 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:49.962 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:49.962 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:49.962 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:49.962 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:49.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd27fc640 10 monclient: _finish_auth 0 2024-04-19T01:46:49.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd27fc640 10 monclient: _check_auth_tickets 2024-04-19T01:46:49.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:46:49.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd27fc640 1 -- 172.21.15.62:0/538233784 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbcc0ab600 con 0x7fcbd40a60b0 2024-04-19T01:46:49.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 5 monclient: authenticate success, global_id 5306 2024-04-19T01:46:49.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:46:49.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd27fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:46:49.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 10 monclient: get_monmap_and_config success 2024-04-19T01:46:49.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 4 set_mon_vals no callback set 2024-04-19T01:46:49.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd27fc640 1 -- 172.21.15.62:0/538233784 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc0ab9d0 con 0x7fcbd40a60b0 2024-04-19T01:46:49.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:49.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:46:49.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd27fc640 10 monclient: dump: 2024-04-19T01:46:49.963 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:49.963 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:49.963 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:49.964 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:49.964 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:49.964 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:49.964 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:49.964 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:49.964 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:49.964 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:49.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 10 monclient: shutdown 2024-04-19T01:46:49.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:46:49.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 1 -- 172.21.15.62:0/538233784 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcbd40a60b0 msgr2=0x7fcbd40a4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:49.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 1 --2- 172.21.15.62:0/538233784 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcbd40a60b0 0x7fcbd40a4ce0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc0097c0 tx=0x7fcbcc01c180 comp rx=0 tx=0).stop 2024-04-19T01:46:49.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 1 -- 172.21.15.62:0/538233784 shutdown_connections 2024-04-19T01:46:49.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 1 --2- 172.21.15.62:0/538233784 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcbd40a60b0 0x7fcbd40a4ce0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:49.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 1 --2- 172.21.15.62:0/538233784 >> v2:172.21.15.62:3300/0 conn(0x7fcbd4137780 0x7fcbd4137b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:49.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 1 -- 172.21.15.62:0/538233784 >> 172.21.15.62:0/538233784 conn(0x7fcbd40aa1d0 msgr2=0x7fcbd40a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:46:49.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 1 -- 172.21.15.62:0/538233784 shutdown_connections 2024-04-19T01:46:49.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 1 -- 172.21.15.62:0/538233784 wait complete. 2024-04-19T01:46:49.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 10 monclient: build_initial_monmap 2024-04-19T01:46:49.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 10 monclient: monmap: 2024-04-19T01:46:49.968 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:46:49.968 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:46:49.968 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:46:49.968 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:46:49.969 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:46:49.969 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:49.969 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:46:49.969 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:46:49.969 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:46:49.969 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:49.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 1 Processor -- start 2024-04-19T01:46:49.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 1 -- start start 2024-04-19T01:46:49.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 10 monclient: init 2024-04-19T01:46:49.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 10 monclient: _reopen_session rank -1 2024-04-19T01:46:49.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-19T01:46:49.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcbd40a60b0 0x7fcbd4136220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:49.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 10 monclient(hunting): picked mon.noname-c con 0x7fcbd40a60b0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:46:49.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcbd4137780 0x7fcbd413c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:49.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 10 monclient(hunting): picked mon.noname-b con 0x7fcbd4137780 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:46:49.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbd41380a0 0x7fcbd413c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:49.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 10 monclient(hunting): picked mon.noname-a con 0x7fcbd41380a0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:46:49.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:49.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd4142170 con 0x7fcbd41380a0 2024-04-19T01:46:49.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:49.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd40a5430 con 0x7fcbd4137780 2024-04-19T01:46:49.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:49.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fcbd41422f0 con 0x7fcbd40a60b0 2024-04-19T01:46:49.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 10 monclient(hunting): _renew_subs 2024-04-19T01:46:49.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbda206640 10 monclient(hunting): authenticate will time out at 1195.787029s 2024-04-19T01:46:49.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbd41380a0 0x7fcbd413c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:49.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbd41380a0 0x7fcbd413c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44150/0 (socket says 172.21.15.62:44150) 2024-04-19T01:46:49.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd3fff640 1 -- 172.21.15.62:0/2667045823 learned_addr learned my addr 172.21.15.62:0/2667045823 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:46:49.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd3fff640 10 monclient(hunting): get_auth_request con 0x7fcbd41380a0 auth_method 0 2024-04-19T01:46:49.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:49.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:49.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:49.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd37fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcbd40a60b0 0x7fcbd4136220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:49.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd37fe640 10 monclient(hunting): get_auth_request con 0x7fcbd40a60b0 auth_method 0 2024-04-19T01:46:49.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:49.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:49.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:49.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:49.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:49.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:49.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:49.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:49.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:49.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd2ffd640 1 --2- 172.21.15.62:0/2667045823 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcbd4137780 0x7fcbd413c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:49.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd3fff640 10 monclient(hunting): handle_auth_done global_id 14772 payload 824 2024-04-19T01:46:49.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd3fff640 1 -- 172.21.15.62:0/2667045823 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcbd40a60b0 msgr2=0x7fcbd4136220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:49.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd3fff640 1 --2- 172.21.15.62:0/2667045823 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcbd40a60b0 0x7fcbd4136220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:49.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd3fff640 1 -- 172.21.15.62:0/2667045823 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcbd4137780 msgr2=0x7fcbd413c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:49.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd3fff640 1 --2- 172.21.15.62:0/2667045823 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcbd4137780 0x7fcbd413c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:49.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd3fff640 10 monclient: _finish_hunting 0 2024-04-19T01:46:49.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd3fff640 1 monclient: found mon.noname-a 2024-04-19T01:46:49.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:46:49.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:46:49.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd3fff640 1 -- 172.21.15.62:0/2667045823 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbd41208f0 con 0x7fcbd41380a0 2024-04-19T01:46:49.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd2ffd640 10 monclient: get_auth_request con 0x7fcbd4137780 auth_method 0 2024-04-19T01:46:49.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd37fe640 1 --2- 172.21.15.62:0/2667045823 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcbd40a60b0 0x7fcbd4136220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:46:49.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.963+0000 7fcbd2ffd640 1 --2- 172.21.15.62:0/2667045823 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcbd4137780 0x7fcbd413c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:46:49.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.967+0000 7fcbd3fff640 1 -- 172.21.15.62:0/2667045823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbd41380a0 msgr2=0x7fcbd413c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:49.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.967+0000 7fcbd3fff640 1 --2- 172.21.15.62:0/2667045823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbd41380a0 0x7fcbd413c6a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc01ca20 tx=0x7fcbcc018f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:49.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.967+0000 7fcbcbfff640 1 -- 172.21.15.62:0/2667045823 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc0b0040 con 0x7fcbd41380a0 2024-04-19T01:46:49.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.967+0000 7fcbcbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:49.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.967+0000 7fcbcbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:46:49.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.967+0000 7fcbcbfff640 10 monclient: dump: 2024-04-19T01:46:49.975 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:49.975 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:49.975 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:49.975 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:49.975 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:49.976 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:49.976 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:49.976 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:49.976 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:49.976 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:49.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.967+0000 7fcbcbfff640 10 monclient: _finish_auth 0 2024-04-19T01:46:49.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.967+0000 7fcbcbfff640 10 monclient: _check_auth_tickets 2024-04-19T01:46:49.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.967+0000 7fcbcbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:46:49.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.967+0000 7fcbcbfff640 1 -- 172.21.15.62:0/2667045823 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbcc0b9070 con 0x7fcbd41380a0 2024-04-19T01:46:49.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.967+0000 7fcbcbfff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:46:49.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.967+0000 7fcbcbfff640 1 -- 172.21.15.62:0/2667045823 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc0aa9b0 con 0x7fcbd41380a0 2024-04-19T01:46:49.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.967+0000 7fcbda206640 5 monclient: authenticate success, global_id 14772 2024-04-19T01:46:49.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.967+0000 7fcbda206640 10 monclient: _renew_subs 2024-04-19T01:46:49.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.967+0000 7fcbda206640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:46:49.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.967+0000 7fcbda206640 1 -- 172.21.15.62:0/2667045823 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbd4120bd0 con 0x7fcbd41380a0 2024-04-19T01:46:49.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.967+0000 7fcbcbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:49.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.967+0000 7fcbcbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:46:49.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.967+0000 7fcbcbfff640 10 monclient: dump: 2024-04-19T01:46:49.977 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:49.977 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:49.977 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:49.977 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:49.977 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:49.977 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:49.977 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:49.977 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:49.977 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:49.977 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:49.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.967+0000 7fcbd17fa640 4 set_mon_vals no callback set 2024-04-19T01:46:49.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.967+0000 7fcbda206640 10 monclient: _renew_subs 2024-04-19T01:46:49.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.967+0000 7fcbda206640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:46:49.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.967+0000 7fcbda206640 1 -- 172.21.15.62:0/2667045823 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbd40a2270 con 0x7fcbd41380a0 2024-04-19T01:46:49.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.967+0000 7fcbda206640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:49.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.967+0000 7fcbda206640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:49.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.967+0000 7fcbda206640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:46:49.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.967+0000 7fcbda206640 1 -- 172.21.15.62:0/2667045823 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbd4068ab0 con 0x7fcbd41380a0 2024-04-19T01:46:49.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.967+0000 7fcbcbfff640 1 -- 172.21.15.62:0/2667045823 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fcbcc0ab4c0 con 0x7fcbd41380a0 2024-04-19T01:46:49.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.967+0000 7fcbcbfff640 1 --2- 172.21.15.62:0/2667045823 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fcbb4036470 0x7fcbb4038930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:49.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.967+0000 7fcbcbfff640 1 -- 172.21.15.62:0/2667045823 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(224..224 src has 108..224) ==== 4738+0+0 (secure 0 0 0) 0x7fcbcc0e31d0 con 0x7fcbd41380a0 2024-04-19T01:46:49.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.967+0000 7fcbd37fe640 1 --2- 172.21.15.62:0/2667045823 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fcbb4036470 0x7fcbb4038930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:49.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.967+0000 7fcbd37fe640 10 monclient: get_auth_request con 0x7fcbb4036470 auth_method 0 2024-04-19T01:46:49.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.971+0000 7fcbd37fe640 1 --2- 172.21.15.62:0/2667045823 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fcbb4036470 0x7fcbb4038930 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4009fd0 tx=0x7fcbc400b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:49.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.975+0000 7fcbcbfff640 1 -- 172.21.15.62:0/2667045823 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fcbd409c2e0 con 0x7fcbd41380a0 2024-04-19T01:46:49.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.975+0000 7fcbcbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:49.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:49.975+0000 7fcbcbfff640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:46:50.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.199+0000 7fcbda206640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:46:50.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.199+0000 7fcbda206640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:46:50.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.199+0000 7fcbda206640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:46:50.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.199+0000 7fcbda206640 1 -- 172.21.15.62:0/2667045823 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcbd4068710 con 0x7fcbd41380a0 2024-04-19T01:46:50.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.199+0000 7fcbcbfff640 1 -- 172.21.15.62:0/2667045823 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fcbcc0bdc50 con 0x7fcbd41380a0 2024-04-19T01:46:50.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.199+0000 7fcbcbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:46:50.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.199+0000 7fcbcbfff640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:46:50.199 INFO:teuthology.orchestra.run.smithi062.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","modified":"2024-04-19T01:41:38.497795Z","created":"2024-04-19T01:41:38.497795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:46:50.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.203+0000 7fcbda206640 1 -- 172.21.15.62:0/2667045823 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fcbb4036470 msgr2=0x7fcbb4038930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:50.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.203+0000 7fcbda206640 1 --2- 172.21.15.62:0/2667045823 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fcbb4036470 0x7fcbb4038930 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4009fd0 tx=0x7fcbc400b040 comp rx=0 tx=0).stop 2024-04-19T01:46:50.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.203+0000 7fcbda206640 10 monclient: shutdown 2024-04-19T01:46:50.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.203+0000 7fcbda206640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:46:50.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.203+0000 7fcbda206640 1 -- 172.21.15.62:0/2667045823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbd41380a0 msgr2=0x7fcbd413c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:50.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.203+0000 7fcbda206640 1 --2- 172.21.15.62:0/2667045823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbd41380a0 0x7fcbd413c6a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc01ca20 tx=0x7fcbcc018f20 comp rx=0 tx=0).stop 2024-04-19T01:46:50.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.203+0000 7fcbda206640 1 -- 172.21.15.62:0/2667045823 shutdown_connections 2024-04-19T01:46:50.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.203+0000 7fcbda206640 1 --2- 172.21.15.62:0/2667045823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbd41380a0 0x7fcbd413c6a0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:50.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.203+0000 7fcbda206640 1 --2- 172.21.15.62:0/2667045823 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcbd4137780 0x7fcbd413c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:50.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.203+0000 7fcbda206640 1 --2- 172.21.15.62:0/2667045823 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fcbb4036470 0x7fcbb4038930 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:50.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.203+0000 7fcbda206640 1 --2- 172.21.15.62:0/2667045823 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcbd40a60b0 0x7fcbd4136220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:50.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.203+0000 7fcbda206640 1 -- 172.21.15.62:0/2667045823 >> 172.21.15.62:0/2667045823 conn(0x7fcbd40aa1d0 msgr2=0x7fcbd40a79d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:46:50.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.203+0000 7fcbda206640 1 -- 172.21.15.62:0/2667045823 shutdown_connections 2024-04-19T01:46:50.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.203+0000 7fcbda206640 1 -- 172.21.15.62:0/2667045823 wait complete. 2024-04-19T01:46:50.217 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-19T01:46:50.217 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-19T01:46:50.217 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-19T01:46:50.217 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-19T01:46:50.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.331+0000 7f2e26ffe640 10 monclient: get_monmap_and_config 2024-04-19T01:46:50.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.331+0000 7f2e26ffe640 10 monclient: build_initial_monmap 2024-04-19T01:46:50.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.331+0000 7f2e26ffe640 10 monclient: monmap: 2024-04-19T01:46:50.333 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:46:50.333 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:50.333 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:46:50.337837+0000 2024-04-19T01:46:50.333 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:46:50.337837+0000 2024-04-19T01:46:50.333 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:46:50.333 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:50.333 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:46:50.333 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:46:50.333 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:46:50.333 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:46:50.333 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:46:50.334 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:50.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e26ffe640 1 Processor -- start 2024-04-19T01:46:50.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e26ffe640 1 -- start start 2024-04-19T01:46:50.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e26ffe640 10 monclient: init 2024-04-19T01:46:50.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e26ffe640 10 monclient: _reopen_session rank -1 2024-04-19T01:46:50.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e26ffe640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-04-19T01:46:50.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e26ffe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2e200a47e0 0x7f2e200a4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:50.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e26ffe640 10 monclient(hunting): picked mon.noname-a con 0x7f2e200a47e0 addr v2:172.21.15.62:3300/0 2024-04-19T01:46:50.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e26ffe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2e200a5100 addr v1:172.21.15.71:6789/0 2024-04-19T01:46:50.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e26ffe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2e201357c0 addr v1:172.21.15.62:6789/0 2024-04-19T01:46:50.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e26ffe640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e2009f7c0 con 0x7f2e201357c0 2024-04-19T01:46:50.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e26ffe640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e2009c2d0 con 0x7f2e200a5100 2024-04-19T01:46:50.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e26ffe640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:50.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e26ffe640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f2e2009ee90 con 0x7f2e200a47e0 2024-04-19T01:46:50.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e26ffe640 10 monclient(hunting): _renew_subs 2024-04-19T01:46:50.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e26ffe640 10 monclient(hunting): authenticate will time out at 1196.156964s 2024-04-19T01:46:50.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e24d9a640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2e200a47e0 0x7f2e200a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:50.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e2559b640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f2e201357c0 0x7f2e2013d600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:39824/0 (socket says 172.21.15.62:39824) 2024-04-19T01:46:50.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e2559b640 1 -- 172.21.15.62:0/1643733216 learned_addr learned my addr 172.21.15.62:0/1643733216 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:46:50.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e24d9a640 10 monclient(hunting): get_auth_request con 0x7f2e200a47e0 auth_method 0 2024-04-19T01:46:50.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e24d9a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:50.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e24d9a640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:50.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e24d9a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:50.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e24d9a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:50.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e24d9a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:50.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e24d9a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:50.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e24d9a640 10 monclient(hunting): handle_auth_done global_id 14775 payload 274 2024-04-19T01:46:50.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e24d9a640 1 -- 172.21.15.62:0/1643733216 >> v1:172.21.15.71:6789/0 conn(0x7f2e200a5100 legacy=0x7f2e201350b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:46:50.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e24d9a640 1 -- 172.21.15.62:0/1643733216 >> v1:172.21.15.62:6789/0 conn(0x7f2e201357c0 legacy=0x7f2e2013d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:46:50.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e24d9a640 10 monclient: _finish_hunting 0 2024-04-19T01:46:50.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e24d9a640 1 monclient: found mon.noname-a 2024-04-19T01:46:50.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e24d9a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:46:50.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e24d9a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:46:50.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.335+0000 7f2e24d9a640 1 -- 172.21.15.62:0/1643733216 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e2009faa0 con 0x7f2e200a47e0 2024-04-19T01:46:50.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e24d9a640 1 -- 172.21.15.62:0/1643733216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e200a47e0 msgr2=0x7f2e200a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:50.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e24d9a640 1 --2- 172.21.15.62:0/1643733216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e200a47e0 0x7f2e200a4bc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2e100097f0 tx=0x7f2e1001c2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c570b9adc1b632e server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:50.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e1f7fe640 1 -- 172.21.15.62:0/1643733216 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e100b0070 con 0x7f2e200a47e0 2024-04-19T01:46:50.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e1f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:50.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e1f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:46:50.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e1f7fe640 10 monclient: dump: 2024-04-19T01:46:50.342 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:50.342 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:50.342 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.342 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.342 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:50.342 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:50.342 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:50.342 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:50.343 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:50.343 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:50.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e1f7fe640 10 monclient: _finish_auth 0 2024-04-19T01:46:50.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e1f7fe640 10 monclient: _check_auth_tickets 2024-04-19T01:46:50.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:46:50.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e1f7fe640 1 -- 172.21.15.62:0/1643733216 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e100ab600 con 0x7f2e200a47e0 2024-04-19T01:46:50.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e1f7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:46:50.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e1f7fe640 1 -- 172.21.15.62:0/1643733216 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e100ab9d0 con 0x7f2e200a47e0 2024-04-19T01:46:50.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e1f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:50.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e1f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:46:50.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e1f7fe640 10 monclient: dump: 2024-04-19T01:46:50.343 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:50.343 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:50.344 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.344 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.344 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:50.344 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:50.344 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:50.344 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:50.344 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:50.344 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:50.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e26ffe640 5 monclient: authenticate success, global_id 14775 2024-04-19T01:46:50.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e26ffe640 10 monclient: get_monmap_and_config success 2024-04-19T01:46:50.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e26ffe640 4 set_mon_vals no callback set 2024-04-19T01:46:50.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e26ffe640 10 monclient: shutdown 2024-04-19T01:46:50.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e26ffe640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:46:50.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e26ffe640 1 -- 172.21.15.62:0/1643733216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e200a47e0 msgr2=0x7f2e200a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:50.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e26ffe640 1 --2- 172.21.15.62:0/1643733216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e200a47e0 0x7f2e200a4bc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2e100097f0 tx=0x7f2e1001c2a0 comp rx=0 tx=0).stop 2024-04-19T01:46:50.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e26ffe640 1 -- 172.21.15.62:0/1643733216 shutdown_connections 2024-04-19T01:46:50.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e26ffe640 1 --2- 172.21.15.62:0/1643733216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e200a47e0 0x7f2e200a4bc0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:50.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e26ffe640 1 -- 172.21.15.62:0/1643733216 >> 172.21.15.62:0/1643733216 conn(0x7f2e200a23d0 msgr2=0x7f2e200a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:46:50.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e26ffe640 1 -- 172.21.15.62:0/1643733216 shutdown_connections 2024-04-19T01:46:50.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e26ffe640 1 -- 172.21.15.62:0/1643733216 wait complete. 2024-04-19T01:46:50.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e26ffe640 10 monclient: build_initial_monmap 2024-04-19T01:46:50.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e26ffe640 10 monclient: monmap: 2024-04-19T01:46:50.345 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:46:50.345 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:46:50.345 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:46:50.345 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:46:50.345 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:46:50.345 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:50.345 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:46:50.345 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:46:50.345 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:46:50.345 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:50.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e26ffe640 1 Processor -- start 2024-04-19T01:46:50.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e26ffe640 1 -- start start 2024-04-19T01:46:50.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e26ffe640 10 monclient: init 2024-04-19T01:46:50.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e26ffe640 10 monclient: _reopen_session rank -1 2024-04-19T01:46:50.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e26ffe640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-19T01:46:50.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e26ffe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2e200a47e0 0x7f2e2013d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:50.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e26ffe640 10 monclient(hunting): picked mon.noname-b con 0x7f2e200a47e0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:46:50.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e26ffe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e200a5100 0x7f2e2013da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:50.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e26ffe640 10 monclient(hunting): picked mon.noname-a con 0x7f2e200a5100 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:46:50.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e26ffe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e201357c0 0x7f2e201cf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:50.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e26ffe640 10 monclient(hunting): picked mon.noname-c con 0x7f2e201357c0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:46:50.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e26ffe640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:50.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.339+0000 7f2e26ffe640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2e2009c8e0 con 0x7f2e200a5100 2024-04-19T01:46:50.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e26ffe640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:50.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e26ffe640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f2e2009c760 con 0x7f2e200a47e0 2024-04-19T01:46:50.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e26ffe640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:50.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e26ffe640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f2e20141540 con 0x7f2e201357c0 2024-04-19T01:46:50.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e26ffe640 10 monclient(hunting): _renew_subs 2024-04-19T01:46:50.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e26ffe640 10 monclient(hunting): authenticate will time out at 1196.164030s 2024-04-19T01:46:50.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e200a5100 0x7f2e2013da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:50.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e2559b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e201357c0 0x7f2e201cf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:50.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e200a5100 0x7f2e2013da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44174/0 (socket says 172.21.15.62:44174) 2024-04-19T01:46:50.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1ffff640 1 -- 172.21.15.62:0/717983263 learned_addr learned my addr 172.21.15.62:0/717983263 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:46:50.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1ffff640 10 monclient(hunting): get_auth_request con 0x7f2e200a5100 auth_method 0 2024-04-19T01:46:50.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:50.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:50.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:50.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e2559b640 10 monclient(hunting): get_auth_request con 0x7f2e201357c0 auth_method 0 2024-04-19T01:46:50.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e2559b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:50.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e2559b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:50.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e2559b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:50.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e2559b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:50.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e2559b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:50.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e2559b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:50.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:50.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:50.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:50.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e24d9a640 1 --2- 172.21.15.62:0/717983263 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2e200a47e0 0x7f2e2013d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:50.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e24d9a640 10 monclient(hunting): get_auth_request con 0x7f2e200a47e0 auth_method 0 2024-04-19T01:46:50.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e24d9a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:50.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e24d9a640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:50.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e24d9a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:50.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1ffff640 10 monclient(hunting): handle_auth_done global_id 14778 payload 824 2024-04-19T01:46:50.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1ffff640 1 -- 172.21.15.62:0/717983263 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e201357c0 msgr2=0x7f2e201cf9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:50.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1ffff640 1 --2- 172.21.15.62:0/717983263 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e201357c0 0x7f2e201cf9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:50.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1ffff640 1 -- 172.21.15.62:0/717983263 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2e200a47e0 msgr2=0x7f2e2013d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:50.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1ffff640 1 --2- 172.21.15.62:0/717983263 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2e200a47e0 0x7f2e2013d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:50.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1ffff640 10 monclient: _finish_hunting 0 2024-04-19T01:46:50.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1ffff640 1 monclient: found mon.noname-a 2024-04-19T01:46:50.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:46:50.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:46:50.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1ffff640 1 -- 172.21.15.62:0/717983263 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e200726d0 con 0x7f2e200a5100 2024-04-19T01:46:50.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e2559b640 1 --2- 172.21.15.62:0/717983263 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f2e201357c0 0x7f2e201cf9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:46:50.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1ffff640 1 -- 172.21.15.62:0/717983263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e200a5100 msgr2=0x7f2e2013da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:50.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1ffff640 1 --2- 172.21.15.62:0/717983263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e200a5100 0x7f2e2013da00 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2e1400c480 tx=0x7f2e1400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:50.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1d7fa640 1 -- 172.21.15.62:0/717983263 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e14012a40 con 0x7f2e200a5100 2024-04-19T01:46:50.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:50.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:46:50.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1d7fa640 10 monclient: dump: 2024-04-19T01:46:50.352 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:50.352 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:50.352 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.352 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.352 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:50.352 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:50.352 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:50.352 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:50.353 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:50.353 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:50.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1d7fa640 10 monclient: _finish_auth 0 2024-04-19T01:46:50.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1d7fa640 10 monclient: _check_auth_tickets 2024-04-19T01:46:50.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:46:50.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1d7fa640 1 -- 172.21.15.62:0/717983263 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e14017070 con 0x7f2e200a5100 2024-04-19T01:46:50.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e26ffe640 5 monclient: authenticate success, global_id 14778 2024-04-19T01:46:50.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e26ffe640 10 monclient: _renew_subs 2024-04-19T01:46:50.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e26ffe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:46:50.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e26ffe640 1 -- 172.21.15.62:0/717983263 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e2013aca0 con 0x7f2e200a5100 2024-04-19T01:46:50.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1d7fa640 10 monclient: handle_config config(0 keys) 2024-04-19T01:46:50.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1d7fa640 1 -- 172.21.15.62:0/717983263 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e14012e40 con 0x7f2e200a5100 2024-04-19T01:46:50.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:50.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:46:50.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1d7fa640 10 monclient: dump: 2024-04-19T01:46:50.354 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:50.354 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:50.354 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.354 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.354 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:50.354 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:50.354 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:50.354 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:50.354 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:50.354 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:50.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e26ffe640 10 monclient: _renew_subs 2024-04-19T01:46:50.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e26ffe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:46:50.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e26ffe640 1 -- 172.21.15.62:0/717983263 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e2013a6e0 con 0x7f2e200a5100 2024-04-19T01:46:50.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e1effd640 4 set_mon_vals no callback set 2024-04-19T01:46:50.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e26ffe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:50.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e26ffe640 1 --2- 172.21.15.62:0/717983263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e2009d040 0x7f2e201cfef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:50.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e26ffe640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:50.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e26ffe640 1 -- 172.21.15.62:0/717983263 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2e200692b0 con 0x7f2e2009d040 2024-04-19T01:46:50.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e24d9a640 1 --2- 172.21.15.62:0/717983263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e2009d040 0x7f2e201cfef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:50.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e24d9a640 10 monclient: get_auth_request con 0x7f2e2009d040 auth_method 0 2024-04-19T01:46:50.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e24d9a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:50.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e24d9a640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:50.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.343+0000 7f2e24d9a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:50.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.347+0000 7f2e24d9a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:50.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.347+0000 7f2e24d9a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:50.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.347+0000 7f2e24d9a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:50.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.347+0000 7f2e1ffff640 1 --2- 172.21.15.62:0/717983263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e200a5100 0x7f2e2013da00 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2e1400c480 tx=0x7f2e1400c940 comp rx=0 tx=0).queue_received will delay after 1.71349e+09 on 0x7f2e140492a0 osd_map(224..224 src has 108..224) 2024-04-19T01:46:50.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.347+0000 7f2e24d9a640 10 monclient(hunting): handle_auth_done global_id 14781 payload 824 2024-04-19T01:46:50.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.347+0000 7f2e1d7fa640 1 -- 172.21.15.62:0/717983263 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f2e1401c460 con 0x7f2e200a5100 2024-04-19T01:46:50.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.347+0000 7f2e24d9a640 1 -- 172.21.15.62:0/717983263 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2e200722c0 con 0x7f2e2009d040 2024-04-19T01:46:50.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.347+0000 7f2e1d7fa640 1 --2- 172.21.15.62:0/717983263 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f2e0c0364a0 0x7f2e0c038960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:50.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.347+0000 7f2e2559b640 1 --2- 172.21.15.62:0/717983263 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f2e0c0364a0 0x7f2e0c038960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:50.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.347+0000 7f2e2559b640 10 monclient: get_auth_request con 0x7f2e0c0364a0 auth_method 0 2024-04-19T01:46:50.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.347+0000 7f2e24d9a640 1 -- 172.21.15.62:0/717983263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e2009d040 msgr2=0x7f2e201cfef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:50.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.347+0000 7f2e24d9a640 1 --2- 172.21.15.62:0/717983263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e2009d040 0x7f2e201cfef0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2e2006dd10 tx=0x7f2e1001aae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:50.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.347+0000 7f2e1d7fa640 1 -- 172.21.15.62:0/717983263 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e100ba030 con 0x7f2e2009d040 2024-04-19T01:46:50.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.347+0000 7f2e1d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:50.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.347+0000 7f2e1d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:46:50.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.347+0000 7f2e1d7fa640 10 monclient: dump: 2024-04-19T01:46:50.357 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:50.357 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:50.357 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.357 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.357 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:50.357 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:50.357 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:50.357 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:50.357 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:50.357 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:50.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.347+0000 7f2e1d7fa640 1 -- 172.21.15.62:0/717983263 <== mon.0 v2:172.21.15.62:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2e100abdb0 con 0x7f2e2009d040 2024-04-19T01:46:50.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.347+0000 7f2e1d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:50.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.347+0000 7f2e1d7fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:46:50.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.347+0000 7f2e1d7fa640 1 -- 172.21.15.62:0/717983263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e2009d040 msgr2=0x7f2e201cfef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:50.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.347+0000 7f2e1d7fa640 1 --2- 172.21.15.62:0/717983263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e2009d040 0x7f2e201cfef0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2e2006dd10 tx=0x7f2e1001aae0 comp rx=0 tx=0).stop 2024-04-19T01:46:50.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.347+0000 7f2e1d7fa640 1 -- 172.21.15.62:0/717983263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e2009d040 msgr2=0x7f2e201cfef0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:46:50.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.347+0000 7f2e1d7fa640 1 --2- 172.21.15.62:0/717983263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e2009d040 0x7f2e201cfef0 secure :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2e2006dd10 tx=0x7f2e1001aae0 comp rx=0 tx=0).stop 2024-04-19T01:46:50.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.347+0000 7f2e2559b640 1 --2- 172.21.15.62:0/717983263 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f2e0c0364a0 0x7f2e0c038960 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2e18005c10 tx=0x7f2e18009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:50.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.355+0000 7f2e26ffe640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:46:50.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.355+0000 7f2e26ffe640 1 --2- 172.21.15.62:0/717983263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e201d0430 0x7f2e20070610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:50.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.355+0000 7f2e26ffe640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:50.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.355+0000 7f2e26ffe640 1 -- 172.21.15.62:0/717983263 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2e201d0810 con 0x7f2e201d0430 2024-04-19T01:46:50.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.355+0000 7f2e24d9a640 1 --2- 172.21.15.62:0/717983263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e201d0430 0x7f2e20070610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:50.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.355+0000 7f2e24d9a640 10 monclient: get_auth_request con 0x7f2e201d0430 auth_method 0 2024-04-19T01:46:50.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.355+0000 7f2e24d9a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:50.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.355+0000 7f2e24d9a640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:50.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.355+0000 7f2e24d9a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:50.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.355+0000 7f2e24d9a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:50.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.355+0000 7f2e24d9a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:50.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.355+0000 7f2e24d9a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:50.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.355+0000 7f2e24d9a640 10 monclient(hunting): handle_auth_done global_id 14784 payload 824 2024-04-19T01:46:50.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.355+0000 7f2e24d9a640 1 -- 172.21.15.62:0/717983263 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2e2013a8e0 con 0x7f2e201d0430 2024-04-19T01:46:50.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.355+0000 7f2e24d9a640 1 -- 172.21.15.62:0/717983263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e201d0430 msgr2=0x7f2e20070610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:50.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.355+0000 7f2e24d9a640 1 --2- 172.21.15.62:0/717983263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e201d0430 0x7f2e20070610 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2e10000c10 tx=0x7f2e10019040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:50.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.359+0000 7f2e1d7fa640 1 -- 172.21.15.62:0/717983263 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e100ba050 con 0x7f2e201d0430 2024-04-19T01:46:50.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.359+0000 7f2e1d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:50.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.359+0000 7f2e1d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:46:50.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.359+0000 7f2e1d7fa640 10 monclient: dump: 2024-04-19T01:46:50.361 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:50.361 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:50.361 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.361 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.361 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:50.361 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:50.361 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:50.361 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:50.361 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:50.361 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:50.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.359+0000 7f2e1d7fa640 1 -- 172.21.15.62:0/717983263 <== mon.0 v2:172.21.15.62:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4719 (secure 0 0 0) 0x7f2e10003e60 con 0x7f2e201d0430 2024-04-19T01:46:50.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.359+0000 7f2e1d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:46:50.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.359+0000 7f2e1d7fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:46:50.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.359+0000 7f2e1d7fa640 1 -- 172.21.15.62:0/717983263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e201d0430 msgr2=0x7f2e20070610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:50.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.359+0000 7f2e1d7fa640 1 --2- 172.21.15.62:0/717983263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e201d0430 0x7f2e20070610 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2e10000c10 tx=0x7f2e10019040 comp rx=0 tx=0).stop 2024-04-19T01:46:50.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.359+0000 7f2e1d7fa640 1 -- 172.21.15.62:0/717983263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e201d0430 msgr2=0x7f2e20070610 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:46:50.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.359+0000 7f2e1d7fa640 1 --2- 172.21.15.62:0/717983263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e201d0430 0x7f2e20070610 secure :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2e10000c10 tx=0x7f2e10019040 comp rx=0 tx=0).stop 2024-04-19T01:46:50.362 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-19T01:46:50.362 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "a", 2024-04-19T01:46:50.362 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 0, 2024-04-19T01:46:50.362 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "leader", 2024-04-19T01:46:50.362 INFO:teuthology.orchestra.run.smithi062.stdout: "election_epoch": 34, 2024-04-19T01:46:50.362 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 153921, 2024-04-19T01:46:50.362 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum": [ 2024-04-19T01:46:50.362 INFO:teuthology.orchestra.run.smithi062.stdout: 0, 2024-04-19T01:46:50.363 INFO:teuthology.orchestra.run.smithi062.stdout: 1, 2024-04-19T01:46:50.363 INFO:teuthology.orchestra.run.smithi062.stdout: 2 2024-04-19T01:46:50.363 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:46:50.363 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_age": 6, 2024-04-19T01:46:50.363 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:46:50.363 INFO:teuthology.orchestra.run.smithi062.stdout: "required_con": "2450521708007866372", 2024-04-19T01:46:50.363 INFO:teuthology.orchestra.run.smithi062.stdout: "required_mon": [ 2024-04-19T01:46:50.363 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:46:50.363 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:46:50.363 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:46:50.363 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:46:50.363 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:46:50.363 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:46:50.363 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:46:50.363 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:46:50.363 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:46:50.364 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:46:50.364 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:46:50.364 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:46:50.364 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:46:50.364 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_mon": [ 2024-04-19T01:46:50.364 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:46:50.364 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:46:50.364 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:46:50.364 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:46:50.364 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:46:50.364 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:46:50.364 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:46:50.364 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:46:50.364 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:46:50.364 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:46:50.364 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:46:50.364 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:50.365 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.365 INFO:teuthology.orchestra.run.smithi062.stdout: "outside_quorum": [], 2024-04-19T01:46:50.365 INFO:teuthology.orchestra.run.smithi062.stdout: "extra_probe_peers": [], 2024-04-19T01:46:50.365 INFO:teuthology.orchestra.run.smithi062.stdout: "sync_provider": [], 2024-04-19T01:46:50.365 INFO:teuthology.orchestra.run.smithi062.stdout: "monmap": { 2024-04-19T01:46:50.365 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 1, 2024-04-19T01:46:50.365 INFO:teuthology.orchestra.run.smithi062.stdout: "fsid": "6466e4b6-2520-485f-95b3-9401915fbbad", 2024-04-19T01:46:50.365 INFO:teuthology.orchestra.run.smithi062.stdout: "modified": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:46:50.365 INFO:teuthology.orchestra.run.smithi062.stdout: "created": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:46:50.365 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release": 19, 2024-04-19T01:46:50.365 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release_name": "squid", 2024-04-19T01:46:50.365 INFO:teuthology.orchestra.run.smithi062.stdout: "election_strategy": 1, 2024-04-19T01:46:50.365 INFO:teuthology.orchestra.run.smithi062.stdout: "disallowed_leaders: ": "", 2024-04-19T01:46:50.365 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false, 2024-04-19T01:46:50.365 INFO:teuthology.orchestra.run.smithi062.stdout: "tiebreaker_mon": "", 2024-04-19T01:46:50.365 INFO:teuthology.orchestra.run.smithi062.stdout: "removed_ranks: ": "", 2024-04-19T01:46:50.365 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:46:50.366 INFO:teuthology.orchestra.run.smithi062.stdout: "persistent": [ 2024-04-19T01:46:50.366 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:46:50.366 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:46:50.366 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:46:50.366 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:46:50.366 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:46:50.366 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:46:50.366 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:46:50.366 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:46:50.366 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:46:50.366 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:46:50.366 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:46:50.366 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:46:50.366 INFO:teuthology.orchestra.run.smithi062.stdout: "optional": [] 2024-04-19T01:46:50.366 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.367 INFO:teuthology.orchestra.run.smithi062.stdout: "mons": [ 2024-04-19T01:46:50.367 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.367 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 0, 2024-04-19T01:46:50.367 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "a", 2024-04-19T01:46:50.367 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:46:50.367 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:46:50.367 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.367 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:46:50.367 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3300", 2024-04-19T01:46:50.367 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:50.367 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.367 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.367 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:46:50.367 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789", 2024-04-19T01:46:50.367 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:50.367 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:50.367 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:50.368 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.368 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789/0", 2024-04-19T01:46:50.368 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6789/0", 2024-04-19T01:46:50.368 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:46:50.368 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:46:50.368 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:46:50.368 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.368 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.368 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 1, 2024-04-19T01:46:50.368 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "b", 2024-04-19T01:46:50.368 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:46:50.368 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:46:50.368 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.368 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:46:50.369 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:3300", 2024-04-19T01:46:50.369 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:50.369 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.369 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.369 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:46:50.369 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789", 2024-04-19T01:46:50.369 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:50.369 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:50.369 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:50.369 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.369 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789/0", 2024-04-19T01:46:50.369 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.71:6789/0", 2024-04-19T01:46:50.369 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:46:50.369 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:46:50.369 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:46:50.369 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.370 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.370 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 2, 2024-04-19T01:46:50.370 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "c", 2024-04-19T01:46:50.370 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:46:50.370 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:46:50.370 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.370 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:46:50.370 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3301", 2024-04-19T01:46:50.370 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:50.370 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.370 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.370 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:46:50.370 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790", 2024-04-19T01:46:50.370 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:50.370 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:50.371 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:50.371 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.371 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790/0", 2024-04-19T01:46:50.371 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6790/0", 2024-04-19T01:46:50.371 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:46:50.371 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:46:50.371 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:46:50.371 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:50.371 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:50.371 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.371 INFO:teuthology.orchestra.run.smithi062.stdout: "feature_map": { 2024-04-19T01:46:50.371 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": [ 2024-04-19T01:46:50.371 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.371 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:46:50.371 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:46:50.372 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:46:50.372 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:50.372 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:46:50.372 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": [ 2024-04-19T01:46:50.372 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.372 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:46:50.372 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:46:50.372 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 2 2024-04-19T01:46:50.372 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:50.372 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:46:50.372 INFO:teuthology.orchestra.run.smithi062.stdout: "client": [ 2024-04-19T01:46:50.372 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.372 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:46:50.372 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:46:50.372 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 5 2024-04-19T01:46:50.373 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:50.373 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:50.373 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.373 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false 2024-04-19T01:46:50.373 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-19T01:46:50.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.359+0000 7f2e26ffe640 1 -- 172.21.15.62:0/717983263 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f2e0c0364a0 msgr2=0x7f2e0c038960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:50.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.359+0000 7f2e26ffe640 1 --2- 172.21.15.62:0/717983263 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f2e0c0364a0 0x7f2e0c038960 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2e18005c10 tx=0x7f2e18009040 comp rx=0 tx=0).stop 2024-04-19T01:46:50.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.359+0000 7f2e26ffe640 10 monclient: shutdown 2024-04-19T01:46:50.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.359+0000 7f2e26ffe640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:46:50.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.359+0000 7f2e26ffe640 1 -- 172.21.15.62:0/717983263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e200a5100 msgr2=0x7f2e2013da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:50.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.359+0000 7f2e26ffe640 1 --2- 172.21.15.62:0/717983263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e200a5100 0x7f2e2013da00 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2e1400c480 tx=0x7f2e1400c940 comp rx=0 tx=0).stop 2024-04-19T01:46:50.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.359+0000 7f2e2559b640 1 -- 172.21.15.62:0/717983263 reap_dead start 2024-04-19T01:46:50.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.359+0000 7f2e1d7fa640 1 -- 172.21.15.62:0/717983263 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(224..224 src has 108..224) ==== 4738+0+0 (unknown 0 0 0) 0x7f2e140492a0 con 0x7f2e200a5100 2024-04-19T01:46:50.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.359+0000 7f2e2559b640 1 -- 172.21.15.62:0/717983263 reap_dead start 2024-04-19T01:46:50.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.359+0000 7f2e26ffe640 1 -- 172.21.15.62:0/717983263 shutdown_connections 2024-04-19T01:46:50.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.359+0000 7f2e26ffe640 1 -- 172.21.15.62:0/717983263 >> 172.21.15.62:0/717983263 conn(0x7f2e200a23d0 msgr2=0x7f2e2006fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:46:50.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.359+0000 7f2e26ffe640 1 -- 172.21.15.62:0/717983263 shutdown_connections 2024-04-19T01:46:50.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.359+0000 7f2e26ffe640 1 -- 172.21.15.62:0/717983263 wait complete. 2024-04-19T01:46:50.375 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-19T01:46:50.375 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-19T01:46:50.375 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-19T01:46:50.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f69572d1640 10 monclient: get_monmap_and_config 2024-04-19T01:46:50.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f69572d1640 10 monclient: build_initial_monmap 2024-04-19T01:46:50.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f69572d1640 10 monclient: monmap: 2024-04-19T01:46:50.493 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:46:50.493 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:50.493 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:46:50.498690+0000 2024-04-19T01:46:50.493 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:46:50.498690+0000 2024-04-19T01:46:50.493 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:46:50.493 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:50.493 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:46:50.493 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:46:50.493 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:46:50.494 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:46:50.494 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:46:50.494 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:50.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f69572d1640 1 Processor -- start 2024-04-19T01:46:50.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f69572d1640 1 -- start start 2024-04-19T01:46:50.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f69572d1640 10 monclient: init 2024-04-19T01:46:50.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f69572d1640 10 monclient: _reopen_session rank -1 2024-04-19T01:46:50.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f69572d1640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-04-19T01:46:50.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f69572d1640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f69500a2ef0 0x7f69501379a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:50.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f69572d1640 10 monclient(hunting): picked mon.noname-a con 0x7f69500a2ef0 addr v2:172.21.15.62:3300/0 2024-04-19T01:46:50.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f69572d1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6950137ee0 0x7f69500a47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:50.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f69572d1640 10 monclient(hunting): picked mon.noname-b con 0x7f6950137ee0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:46:50.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f69572d1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f69500a4d20 addr v1:172.21.15.62:6789/0 2024-04-19T01:46:50.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f69572d1640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6950137550 con 0x7f69500a4d20 2024-04-19T01:46:50.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f69572d1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:50.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f69572d1640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f695009bd90 con 0x7f69500a2ef0 2024-04-19T01:46:50.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f69572d1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:50.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f69572d1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f695009c070 con 0x7f6950137ee0 2024-04-19T01:46:50.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f69572d1640 10 monclient(hunting): _renew_subs 2024-04-19T01:46:50.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f69572d1640 10 monclient(hunting): authenticate will time out at 1196.318707s 2024-04-19T01:46:50.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f695486c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6950137ee0 0x7f69500a47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:50.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f695586e640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f69500a4d20 0x7f695013e590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:39832/0 (socket says 172.21.15.62:39832) 2024-04-19T01:46:50.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f695586e640 1 -- 172.21.15.62:0/514702709 learned_addr learned my addr 172.21.15.62:0/514702709 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:46:50.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f695506d640 1 --2- 172.21.15.62:0/514702709 >> v2:172.21.15.62:3300/0 conn(0x7f69500a2ef0 0x7f69501379a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:50.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f695486c640 10 monclient(hunting): get_auth_request con 0x7f6950137ee0 auth_method 0 2024-04-19T01:46:50.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f695486c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:50.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f695486c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:50.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f695486c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:50.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f695506d640 10 monclient(hunting): get_auth_request con 0x7f69500a2ef0 auth_method 0 2024-04-19T01:46:50.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f695506d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:50.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f695506d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:50.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f695506d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:50.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f695486c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:50.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f695486c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:50.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f695486c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:50.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f695506d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:50.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f695506d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:50.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f695506d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:50.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f695486c640 10 monclient(hunting): handle_auth_done global_id 5315 payload 274 2024-04-19T01:46:50.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f695486c640 1 -- 172.21.15.62:0/514702709 >> v2:172.21.15.62:3300/0 conn(0x7f69500a2ef0 msgr2=0x7f69501379a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:46:50.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f695486c640 1 --2- 172.21.15.62:0/514702709 >> v2:172.21.15.62:3300/0 conn(0x7f69500a2ef0 0x7f69501379a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:50.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f695486c640 1 -- 172.21.15.62:0/514702709 >> v1:172.21.15.62:6789/0 conn(0x7f69500a4d20 legacy=0x7f695013e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:46:50.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f695486c640 10 monclient: _finish_hunting 0 2024-04-19T01:46:50.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f695486c640 1 monclient: found mon.noname-b 2024-04-19T01:46:50.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f695486c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:46:50.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f695486c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:46:50.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f695486c640 1 -- 172.21.15.62:0/514702709 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f695009c350 con 0x7f6950137ee0 2024-04-19T01:46:50.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.495+0000 7f695506d640 1 --2- 172.21.15.62:0/514702709 >> v2:172.21.15.62:3300/0 conn(0x7f69500a2ef0 0x7f69501379a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:46:50.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.503+0000 7f695486c640 1 -- 172.21.15.62:0/514702709 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6950137ee0 msgr2=0x7f69500a47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:50.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.503+0000 7f695486c640 1 --2- 172.21.15.62:0/514702709 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6950137ee0 0x7f69500a47e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f69440097c0 tx=0x7f694401c300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4989b520b97f4eec server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:50.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.503+0000 7f694ffff640 1 -- 172.21.15.62:0/514702709 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69440b0070 con 0x7f6950137ee0 2024-04-19T01:46:50.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.503+0000 7f694ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:50.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.503+0000 7f694ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:46:50.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.503+0000 7f694ffff640 10 monclient: dump: 2024-04-19T01:46:50.503 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:50.504 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:50.504 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.504 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.504 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:50.504 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:50.504 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:50.504 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:50.504 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:50.504 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:50.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.503+0000 7f694ffff640 10 monclient: _finish_auth 0 2024-04-19T01:46:50.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.503+0000 7f694ffff640 10 monclient: _check_auth_tickets 2024-04-19T01:46:50.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.503+0000 7f694ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:46:50.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.503+0000 7f694ffff640 1 -- 172.21.15.62:0/514702709 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69440ab600 con 0x7f6950137ee0 2024-04-19T01:46:50.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.503+0000 7f694ffff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:46:50.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.503+0000 7f694ffff640 1 -- 172.21.15.62:0/514702709 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69440ab9d0 con 0x7f6950137ee0 2024-04-19T01:46:50.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.503+0000 7f694ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:50.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.503+0000 7f694ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:46:50.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.503+0000 7f694ffff640 10 monclient: dump: 2024-04-19T01:46:50.505 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:50.505 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:50.505 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.505 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.505 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:50.505 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:50.505 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:50.505 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:50.505 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:50.505 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:50.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.503+0000 7f69572d1640 5 monclient: authenticate success, global_id 5315 2024-04-19T01:46:50.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.503+0000 7f69572d1640 10 monclient: get_monmap_and_config success 2024-04-19T01:46:50.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.503+0000 7f69572d1640 4 set_mon_vals no callback set 2024-04-19T01:46:50.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f69572d1640 10 monclient: shutdown 2024-04-19T01:46:50.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f69572d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:46:50.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f69572d1640 1 -- 172.21.15.62:0/514702709 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6950137ee0 msgr2=0x7f69500a47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:50.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f69572d1640 1 --2- 172.21.15.62:0/514702709 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6950137ee0 0x7f69500a47e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f69440097c0 tx=0x7f694401c300 comp rx=0 tx=0).stop 2024-04-19T01:46:50.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f69572d1640 1 -- 172.21.15.62:0/514702709 shutdown_connections 2024-04-19T01:46:50.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f69572d1640 1 --2- 172.21.15.62:0/514702709 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6950137ee0 0x7f69500a47e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:50.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f69572d1640 1 --2- 172.21.15.62:0/514702709 >> v2:172.21.15.62:3300/0 conn(0x7f69500a2ef0 0x7f69501379a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:50.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f69572d1640 1 -- 172.21.15.62:0/514702709 >> 172.21.15.62:0/514702709 conn(0x7f69500a23d0 msgr2=0x7f69500a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:46:50.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f69572d1640 1 -- 172.21.15.62:0/514702709 shutdown_connections 2024-04-19T01:46:50.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f69572d1640 1 -- 172.21.15.62:0/514702709 wait complete. 2024-04-19T01:46:50.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f69572d1640 10 monclient: build_initial_monmap 2024-04-19T01:46:50.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f69572d1640 10 monclient: monmap: 2024-04-19T01:46:50.509 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:46:50.510 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:46:50.510 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:46:50.510 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:46:50.510 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:46:50.510 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:50.510 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:46:50.510 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:46:50.510 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:46:50.510 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:50.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f69572d1640 1 Processor -- start 2024-04-19T01:46:50.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f69572d1640 1 -- start start 2024-04-19T01:46:50.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f69572d1640 10 monclient: init 2024-04-19T01:46:50.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f69572d1640 10 monclient: _reopen_session rank -1 2024-04-19T01:46:50.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f69572d1640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-19T01:46:50.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f69572d1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69500a2ef0 0x7f6950136020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:50.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f69572d1640 10 monclient(hunting): picked mon.noname-b con 0x7f69500a2ef0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:46:50.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f69572d1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69500a4d20 0x7f695011cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:50.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f69572d1640 10 monclient(hunting): picked mon.noname-a con 0x7f69500a4d20 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:46:50.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f69572d1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6950137ee0 0x7f695011d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:50.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f69572d1640 10 monclient(hunting): picked mon.noname-c con 0x7f6950137ee0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:46:50.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f69572d1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:50.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f69572d1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f69501424d0 con 0x7f69500a4d20 2024-04-19T01:46:50.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f69572d1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:50.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f69572d1640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6950072520 con 0x7f69500a2ef0 2024-04-19T01:46:50.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f69572d1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:50.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f69572d1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f6950142650 con 0x7f6950137ee0 2024-04-19T01:46:50.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f69572d1640 10 monclient(hunting): _renew_subs 2024-04-19T01:46:50.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f69572d1640 10 monclient(hunting): authenticate will time out at 1196.330298s 2024-04-19T01:46:50.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f695586e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6950137ee0 0x7f695011d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:50.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f695486c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69500a4d20 0x7f695011cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:50.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f695586e640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6950137ee0 0x7f695011d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:53594/0 (socket says 172.21.15.62:53594) 2024-04-19T01:46:50.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f695586e640 1 -- 172.21.15.62:0/3725818634 learned_addr learned my addr 172.21.15.62:0/3725818634 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:46:50.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f695486c640 1 --2- 172.21.15.62:0/3725818634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69500a4d20 0x7f695011cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44220/0 (socket says 172.21.15.62:44220) 2024-04-19T01:46:50.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f695586e640 10 monclient(hunting): get_auth_request con 0x7f6950137ee0 auth_method 0 2024-04-19T01:46:50.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f695586e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:50.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f695586e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:50.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f695586e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:50.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f695486c640 10 monclient(hunting): get_auth_request con 0x7f69500a4d20 auth_method 0 2024-04-19T01:46:50.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f695486c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:50.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f695486c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:50.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f695486c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:50.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f695486c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:50.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f695486c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:50.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f695486c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:50.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f695586e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:50.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f695586e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:50.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f695586e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:50.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f695486c640 10 monclient(hunting): handle_auth_done global_id 14790 payload 824 2024-04-19T01:46:50.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f695486c640 1 -- 172.21.15.62:0/3725818634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6950137ee0 msgr2=0x7f695011d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:50.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f695486c640 1 --2- 172.21.15.62:0/3725818634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6950137ee0 0x7f695011d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:50.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f695486c640 1 -- 172.21.15.62:0/3725818634 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69500a2ef0 msgr2=0x7f6950136020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:50.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f695486c640 1 --2- 172.21.15.62:0/3725818634 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69500a2ef0 0x7f6950136020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:50.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f695486c640 10 monclient: _finish_hunting 0 2024-04-19T01:46:50.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f695486c640 1 monclient: found mon.noname-a 2024-04-19T01:46:50.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f695486c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:46:50.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f695486c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:46:50.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f695486c640 1 -- 172.21.15.62:0/3725818634 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f695011f220 con 0x7f69500a4d20 2024-04-19T01:46:50.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.507+0000 7f695586e640 1 --2- 172.21.15.62:0/3725818634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6950137ee0 0x7f695011d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:46:50.514 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30486]: starting. 2024-04-19T01:46:50.514 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30486]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-30484 2024-04-19T01:46:50.514 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30486]: shutting down. 2024-04-19T01:46:50.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f695486c640 1 -- 172.21.15.62:0/3725818634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69500a4d20 msgr2=0x7f695011cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:50.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f695486c640 1 --2- 172.21.15.62:0/3725818634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69500a4d20 0x7f695011cac0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f69440ad000 tx=0x7f6944019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:50.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f694dffb640 1 -- 172.21.15.62:0/3725818634 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69440ba020 con 0x7f69500a4d20 2024-04-19T01:46:50.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f694dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:50.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f694dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:46:50.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f694dffb640 10 monclient: dump: 2024-04-19T01:46:50.518 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:50.518 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:50.518 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.518 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.519 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:50.519 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:50.519 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:50.519 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:50.519 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:50.519 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:50.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f694dffb640 10 monclient: _finish_auth 0 2024-04-19T01:46:50.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f694dffb640 10 monclient: _check_auth_tickets 2024-04-19T01:46:50.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f694dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:46:50.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f694dffb640 1 -- 172.21.15.62:0/3725818634 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69440b68a0 con 0x7f69500a4d20 2024-04-19T01:46:50.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f69572d1640 5 monclient: authenticate success, global_id 14790 2024-04-19T01:46:50.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f69572d1640 10 monclient: _renew_subs 2024-04-19T01:46:50.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f69572d1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:46:50.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f69572d1640 1 -- 172.21.15.62:0/3725818634 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f695013f350 con 0x7f69500a4d20 2024-04-19T01:46:50.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f694dffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:46:50.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f694dffb640 1 -- 172.21.15.62:0/3725818634 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69440b0040 con 0x7f69500a4d20 2024-04-19T01:46:50.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f694dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:50.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f694dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:46:50.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f694dffb640 10 monclient: dump: 2024-04-19T01:46:50.520 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:50.520 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:50.520 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.520 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.520 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:50.520 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:50.520 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:50.520 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:50.520 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:50.521 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:50.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f69572d1640 10 monclient: _renew_subs 2024-04-19T01:46:50.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f69572d1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:46:50.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f69572d1640 1 -- 172.21.15.62:0/3725818634 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f695011efa0 con 0x7f69500a4d20 2024-04-19T01:46:50.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f694f7fe640 4 set_mon_vals no callback set 2024-04-19T01:46:50.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f69572d1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:50.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f69572d1640 1 --2- 172.21.15.62:0/3725818634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f695002c450 0x7f695002c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:50.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f69572d1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:50.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f69572d1640 1 -- 172.21.15.62:0/3725818634 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f69500692b0 con 0x7f695002c450 2024-04-19T01:46:50.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f694dffb640 1 -- 172.21.15.62:0/3725818634 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f69440bd020 con 0x7f69500a4d20 2024-04-19T01:46:50.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f695506d640 1 --2- 172.21.15.62:0/3725818634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f695002c450 0x7f695002c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:50.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f695506d640 10 monclient: get_auth_request con 0x7f695002c450 auth_method 0 2024-04-19T01:46:50.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f695506d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:50.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f695506d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:50.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f695506d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:50.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f695506d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:50.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f695506d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:50.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f695506d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:50.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f694dffb640 1 --2- 172.21.15.62:0/3725818634 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f693c036420 0x7f693c0388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:50.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f694dffb640 1 -- 172.21.15.62:0/3725818634 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(225..225 src has 108..225) ==== 4346+0+0 (secure 0 0 0) 0x7f69440e3200 con 0x7f69500a4d20 2024-04-19T01:46:50.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f695506d640 10 monclient(hunting): handle_auth_done global_id 5321 payload 824 2024-04-19T01:46:50.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f695506d640 1 -- 172.21.15.62:0/3725818634 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f695013ba00 con 0x7f695002c450 2024-04-19T01:46:50.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f695586e640 1 --2- 172.21.15.62:0/3725818634 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f693c036420 0x7f693c0388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:50.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.511+0000 7f695586e640 10 monclient: get_auth_request con 0x7f693c036420 auth_method 0 2024-04-19T01:46:50.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.515+0000 7f695506d640 1 -- 172.21.15.62:0/3725818634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f695002c450 msgr2=0x7f695002c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:50.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.515+0000 7f695506d640 1 --2- 172.21.15.62:0/3725818634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f695002c450 0x7f695002c830 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f694000ec30 tx=0x7f694000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:50.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.515+0000 7f694dffb640 1 -- 172.21.15.62:0/3725818634 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6940018020 con 0x7f695002c450 2024-04-19T01:46:50.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.515+0000 7f694dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:50.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.515+0000 7f694dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:46:50.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.515+0000 7f694dffb640 10 monclient: dump: 2024-04-19T01:46:50.523 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:50.523 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:50.523 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.523 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.523 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:50.523 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:50.523 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:50.523 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:50.523 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:50.523 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:50.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.515+0000 7f694dffb640 1 -- 172.21.15.62:0/3725818634 <== mon.2 v2:172.21.15.62:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6940014680 con 0x7f695002c450 2024-04-19T01:46:50.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.515+0000 7f694dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:50.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.515+0000 7f694dffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:46:50.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.515+0000 7f694dffb640 1 -- 172.21.15.62:0/3725818634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f695002c450 msgr2=0x7f695002c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:50.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.515+0000 7f694dffb640 1 --2- 172.21.15.62:0/3725818634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f695002c450 0x7f695002c830 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f694000ec30 tx=0x7f694000c490 comp rx=0 tx=0).stop 2024-04-19T01:46:50.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.515+0000 7f694dffb640 1 -- 172.21.15.62:0/3725818634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f695002c450 msgr2=0x7f695002c830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:46:50.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.515+0000 7f694dffb640 1 --2- 172.21.15.62:0/3725818634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f695002c450 0x7f695002c830 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:50.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.515+0000 7f695586e640 1 --2- 172.21.15.62:0/3725818634 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f693c036420 0x7f693c0388e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6948007d60 tx=0x7f6948007cb0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:50.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.519+0000 7f69572d1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:46:50.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.519+0000 7f69572d1640 1 --2- 172.21.15.62:0/3725818634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f695002cd70 0x7f695002d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:50.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.519+0000 7f69572d1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:50.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.519+0000 7f69572d1640 1 -- 172.21.15.62:0/3725818634 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f69501d0350 con 0x7f695002cd70 2024-04-19T01:46:50.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.519+0000 7f695506d640 1 --2- 172.21.15.62:0/3725818634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f695002cd70 0x7f695002d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:50.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.519+0000 7f695506d640 10 monclient: get_auth_request con 0x7f695002cd70 auth_method 0 2024-04-19T01:46:50.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.519+0000 7f695506d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:50.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.519+0000 7f695506d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:50.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.519+0000 7f695506d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:50.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.523+0000 7f695506d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:50.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.523+0000 7f695506d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:50.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.523+0000 7f695506d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:50.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.523+0000 7f695506d640 10 monclient(hunting): handle_auth_done global_id 5324 payload 824 2024-04-19T01:46:50.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.523+0000 7f695506d640 1 -- 172.21.15.62:0/3725818634 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f695011dd00 con 0x7f695002cd70 2024-04-19T01:46:50.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.523+0000 7f695506d640 1 -- 172.21.15.62:0/3725818634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f695002cd70 msgr2=0x7f695002d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:50.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.523+0000 7f695506d640 1 --2- 172.21.15.62:0/3725818634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f695002cd70 0x7f695002d180 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6940019000 tx=0x7f6940004a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:50.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.523+0000 7f694dffb640 1 -- 172.21.15.62:0/3725818634 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6940018050 con 0x7f695002cd70 2024-04-19T01:46:50.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.523+0000 7f694dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:50.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.523+0000 7f694dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:46:50.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.523+0000 7f694dffb640 10 monclient: dump: 2024-04-19T01:46:50.527 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:50.527 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:50.527 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.527 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.527 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:50.527 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:50.527 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:50.527 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:50.528 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:50.528 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:50.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.523+0000 7f694dffb640 1 -- 172.21.15.62:0/3725818634 <== mon.2 v2:172.21.15.62:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f6940014e30 con 0x7f695002cd70 2024-04-19T01:46:50.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.523+0000 7f694dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:46:50.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.523+0000 7f694dffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:46:50.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.523+0000 7f694dffb640 1 -- 172.21.15.62:0/3725818634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f695002cd70 msgr2=0x7f695002d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:50.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.523+0000 7f694dffb640 1 --2- 172.21.15.62:0/3725818634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f695002cd70 0x7f695002d180 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6940019000 tx=0x7f6940004a50 comp rx=0 tx=0).stop 2024-04-19T01:46:50.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.523+0000 7f694dffb640 1 -- 172.21.15.62:0/3725818634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f695002cd70 msgr2=0x7f695002d180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:46:50.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.523+0000 7f694dffb640 1 --2- 172.21.15.62:0/3725818634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f695002cd70 0x7f695002d180 secure :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6940019000 tx=0x7f6940004a50 comp rx=0 tx=0).stop 2024-04-19T01:46:50.528 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-19T01:46:50.528 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "c", 2024-04-19T01:46:50.528 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 2, 2024-04-19T01:46:50.528 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "peon", 2024-04-19T01:46:50.529 INFO:teuthology.orchestra.run.smithi062.stdout: "election_epoch": 34, 2024-04-19T01:46:50.529 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 286728, 2024-04-19T01:46:50.529 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum": [ 2024-04-19T01:46:50.529 INFO:teuthology.orchestra.run.smithi062.stdout: 0, 2024-04-19T01:46:50.529 INFO:teuthology.orchestra.run.smithi062.stdout: 1, 2024-04-19T01:46:50.529 INFO:teuthology.orchestra.run.smithi062.stdout: 2 2024-04-19T01:46:50.529 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:46:50.529 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_age": 6, 2024-04-19T01:46:50.529 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:46:50.529 INFO:teuthology.orchestra.run.smithi062.stdout: "required_con": "2450521708007866372", 2024-04-19T01:46:50.529 INFO:teuthology.orchestra.run.smithi062.stdout: "required_mon": [ 2024-04-19T01:46:50.529 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:46:50.529 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:46:50.529 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:46:50.529 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:46:50.529 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:46:50.530 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:46:50.530 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:46:50.530 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:46:50.530 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:46:50.530 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:46:50.530 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:46:50.530 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:46:50.530 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:46:50.530 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_mon": [ 2024-04-19T01:46:50.530 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:46:50.530 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:46:50.530 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:46:50.530 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:46:50.530 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:46:50.531 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:46:50.531 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:46:50.531 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:46:50.531 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:46:50.531 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:46:50.531 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:46:50.531 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:50.531 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.531 INFO:teuthology.orchestra.run.smithi062.stdout: "outside_quorum": [], 2024-04-19T01:46:50.531 INFO:teuthology.orchestra.run.smithi062.stdout: "extra_probe_peers": [], 2024-04-19T01:46:50.531 INFO:teuthology.orchestra.run.smithi062.stdout: "sync_provider": [], 2024-04-19T01:46:50.531 INFO:teuthology.orchestra.run.smithi062.stdout: "monmap": { 2024-04-19T01:46:50.531 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 1, 2024-04-19T01:46:50.531 INFO:teuthology.orchestra.run.smithi062.stdout: "fsid": "6466e4b6-2520-485f-95b3-9401915fbbad", 2024-04-19T01:46:50.532 INFO:teuthology.orchestra.run.smithi062.stdout: "modified": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:46:50.532 INFO:teuthology.orchestra.run.smithi062.stdout: "created": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:46:50.532 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release": 19, 2024-04-19T01:46:50.532 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release_name": "squid", 2024-04-19T01:46:50.532 INFO:teuthology.orchestra.run.smithi062.stdout: "election_strategy": 1, 2024-04-19T01:46:50.532 INFO:teuthology.orchestra.run.smithi062.stdout: "disallowed_leaders: ": "", 2024-04-19T01:46:50.532 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false, 2024-04-19T01:46:50.532 INFO:teuthology.orchestra.run.smithi062.stdout: "tiebreaker_mon": "", 2024-04-19T01:46:50.532 INFO:teuthology.orchestra.run.smithi062.stdout: "removed_ranks: ": "", 2024-04-19T01:46:50.532 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:46:50.532 INFO:teuthology.orchestra.run.smithi062.stdout: "persistent": [ 2024-04-19T01:46:50.532 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:46:50.532 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:46:50.532 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:46:50.532 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:46:50.532 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:46:50.532 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:46:50.533 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:46:50.533 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:46:50.533 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:46:50.533 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:46:50.533 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:46:50.533 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:46:50.533 INFO:teuthology.orchestra.run.smithi062.stdout: "optional": [] 2024-04-19T01:46:50.533 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.533 INFO:teuthology.orchestra.run.smithi062.stdout: "mons": [ 2024-04-19T01:46:50.533 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.533 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 0, 2024-04-19T01:46:50.533 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "a", 2024-04-19T01:46:50.533 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:46:50.533 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:46:50.533 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.533 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:46:50.534 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3300", 2024-04-19T01:46:50.534 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:50.534 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.534 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.534 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:46:50.534 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789", 2024-04-19T01:46:50.534 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:50.534 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:50.534 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:50.534 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.534 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789/0", 2024-04-19T01:46:50.534 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6789/0", 2024-04-19T01:46:50.534 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:46:50.534 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:46:50.535 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:46:50.535 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.535 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.535 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 1, 2024-04-19T01:46:50.535 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "b", 2024-04-19T01:46:50.535 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:46:50.535 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:46:50.535 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.535 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:46:50.535 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:3300", 2024-04-19T01:46:50.535 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:50.535 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.535 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.535 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:46:50.535 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789", 2024-04-19T01:46:50.535 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:50.535 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:50.536 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:50.536 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.536 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789/0", 2024-04-19T01:46:50.536 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.71:6789/0", 2024-04-19T01:46:50.536 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:46:50.536 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:46:50.536 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:46:50.536 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.536 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.536 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 2, 2024-04-19T01:46:50.536 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "c", 2024-04-19T01:46:50.536 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:46:50.536 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:46:50.536 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.536 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:46:50.537 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3301", 2024-04-19T01:46:50.537 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:50.537 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.537 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.537 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:46:50.537 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790", 2024-04-19T01:46:50.537 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:50.537 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:50.537 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:50.537 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.537 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790/0", 2024-04-19T01:46:50.537 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6790/0", 2024-04-19T01:46:50.537 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:46:50.537 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:46:50.538 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:46:50.538 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:50.538 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:50.538 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.538 INFO:teuthology.orchestra.run.smithi062.stdout: "feature_map": { 2024-04-19T01:46:50.538 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": [ 2024-04-19T01:46:50.538 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.538 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:46:50.538 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:46:50.538 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:46:50.538 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:50.538 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:46:50.538 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": [ 2024-04-19T01:46:50.539 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.539 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:46:50.539 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:46:50.539 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 4 2024-04-19T01:46:50.539 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:50.539 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:46:50.539 INFO:teuthology.orchestra.run.smithi062.stdout: "client": [ 2024-04-19T01:46:50.539 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.539 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:46:50.539 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:46:50.539 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 3 2024-04-19T01:46:50.539 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:50.539 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:46:50.539 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": [ 2024-04-19T01:46:50.539 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.540 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:46:50.540 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:46:50.540 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:46:50.540 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:50.540 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:50.540 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.540 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false 2024-04-19T01:46:50.540 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-19T01:46:50.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.523+0000 7f69572d1640 1 -- 172.21.15.62:0/3725818634 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f693c036420 msgr2=0x7f693c0388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:50.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.523+0000 7f69572d1640 1 --2- 172.21.15.62:0/3725818634 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f693c036420 0x7f693c0388e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6948007d60 tx=0x7f6948007cb0 comp rx=0 tx=0).stop 2024-04-19T01:46:50.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.523+0000 7f69572d1640 10 monclient: shutdown 2024-04-19T01:46:50.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.523+0000 7f69572d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:46:50.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.523+0000 7f69572d1640 1 -- 172.21.15.62:0/3725818634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69500a4d20 msgr2=0x7f695011cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:50.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.523+0000 7f69572d1640 1 --2- 172.21.15.62:0/3725818634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69500a4d20 0x7f695011cac0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f69440ad000 tx=0x7f6944019100 comp rx=0 tx=0).stop 2024-04-19T01:46:50.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.523+0000 7f695586e640 1 -- 172.21.15.62:0/3725818634 reap_dead start 2024-04-19T01:46:50.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.523+0000 7f695586e640 1 -- 172.21.15.62:0/3725818634 reap_dead start 2024-04-19T01:46:50.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.523+0000 7f69572d1640 1 -- 172.21.15.62:0/3725818634 shutdown_connections 2024-04-19T01:46:50.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.523+0000 7f69572d1640 1 -- 172.21.15.62:0/3725818634 >> 172.21.15.62:0/3725818634 conn(0x7f69500a23d0 msgr2=0x7f69500a0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:46:50.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.523+0000 7f69572d1640 1 -- 172.21.15.62:0/3725818634 shutdown_connections 2024-04-19T01:46:50.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.523+0000 7f69572d1640 1 -- 172.21.15.62:0/3725818634 wait complete. 2024-04-19T01:46:50.542 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-19T01:46:50.543 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-19T01:46:50.543 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T01:46:50.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d7060640 10 monclient: get_monmap_and_config 2024-04-19T01:46:50.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d7060640 10 monclient: build_initial_monmap 2024-04-19T01:46:50.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d7060640 10 monclient: monmap: 2024-04-19T01:46:50.657 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:46:50.657 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:50.657 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:46:50.663010+0000 2024-04-19T01:46:50.658 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:46:50.663010+0000 2024-04-19T01:46:50.658 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:46:50.658 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:50.658 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:46:50.658 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:46:50.658 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:46:50.658 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:46:50.658 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:46:50.658 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:50.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d7060640 1 Processor -- start 2024-04-19T01:46:50.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d7060640 1 -- start start 2024-04-19T01:46:50.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d7060640 10 monclient: init 2024-04-19T01:46:50.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d7060640 10 monclient: _reopen_session rank -1 2024-04-19T01:46:50.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d7060640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-04-19T01:46:50.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d7060640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa4d0137780 0x7fa4d0137b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:50.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d7060640 10 monclient(hunting): picked mon.noname-a con 0x7fa4d0137780 addr v2:172.21.15.62:3300/0 2024-04-19T01:46:50.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d7060640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa4d01380a0 0x7fa4d00a45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:50.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d7060640 10 monclient(hunting): picked mon.noname-c con 0x7fa4d01380a0 addr v2:172.21.15.71:3300/0 2024-04-19T01:46:50.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d7060640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa4d00a60b0 addr v1:172.21.15.62:6789/0 2024-04-19T01:46:50.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d7060640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4d009c2e0 con 0x7fa4d00a60b0 2024-04-19T01:46:50.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d7060640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:50.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d7060640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fa4d009bd20 con 0x7fa4d0137780 2024-04-19T01:46:50.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d7060640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:50.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d7060640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fa4d009c000 con 0x7fa4d01380a0 2024-04-19T01:46:50.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d7060640 10 monclient(hunting): _renew_subs 2024-04-19T01:46:50.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d7060640 10 monclient(hunting): authenticate will time out at 1196.482553s 2024-04-19T01:46:50.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d4dfc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa4d0137780 0x7fa4d0137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:50.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d4dfc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa4d0137780 0x7fa4d0137b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44232/0 (socket says 172.21.15.62:44232) 2024-04-19T01:46:50.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d55fd640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fa4d00a60b0 0x7fa4d00a4b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:39840/0 (socket says 172.21.15.62:39840) 2024-04-19T01:46:50.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d4dfc640 1 -- 172.21.15.62:0/4028414772 learned_addr learned my addr 172.21.15.62:0/4028414772 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:46:50.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d4dfc640 10 monclient(hunting): get_auth_request con 0x7fa4d0137780 auth_method 0 2024-04-19T01:46:50.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d4dfc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:50.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d4dfc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:50.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d4dfc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:50.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d4dfc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:50.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d4dfc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:50.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d4dfc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:50.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4cffff640 1 --2- 172.21.15.62:0/4028414772 >> v2:172.21.15.71:3300/0 conn(0x7fa4d01380a0 0x7fa4d00a45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:50.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4cffff640 10 monclient(hunting): get_auth_request con 0x7fa4d01380a0 auth_method 0 2024-04-19T01:46:50.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:50.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4cffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:50.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4cffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:50.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d4dfc640 10 monclient(hunting): handle_auth_done global_id 14802 payload 274 2024-04-19T01:46:50.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d4dfc640 1 -- 172.21.15.62:0/4028414772 >> v2:172.21.15.71:3300/0 conn(0x7fa4d01380a0 msgr2=0x7fa4d00a45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:46:50.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d4dfc640 1 --2- 172.21.15.62:0/4028414772 >> v2:172.21.15.71:3300/0 conn(0x7fa4d01380a0 0x7fa4d00a45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:50.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d4dfc640 1 -- 172.21.15.62:0/4028414772 >> v1:172.21.15.62:6789/0 conn(0x7fa4d00a60b0 legacy=0x7fa4d00a4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:46:50.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d4dfc640 10 monclient: _finish_hunting 0 2024-04-19T01:46:50.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d4dfc640 1 monclient: found mon.noname-a 2024-04-19T01:46:50.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d4dfc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:46:50.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d4dfc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:46:50.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.659+0000 7fa4d4dfc640 1 -- 172.21.15.62:0/4028414772 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4d0073210 con 0x7fa4d0137780 2024-04-19T01:46:50.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.663+0000 7fa4d4dfc640 1 -- 172.21.15.62:0/4028414772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d0137780 msgr2=0x7fa4d0137b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:50.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.663+0000 7fa4d4dfc640 1 --2- 172.21.15.62:0/4028414772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d0137780 0x7fa4d0137b60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa4d00a74f0 tx=0x7fa4c001c200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecfc9736c2685e7a server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:50.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.663+0000 7fa4cf7fe640 1 -- 172.21.15.62:0/4028414772 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c00b0070 con 0x7fa4d0137780 2024-04-19T01:46:50.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.663+0000 7fa4cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:50.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.663+0000 7fa4cf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:46:50.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.663+0000 7fa4cf7fe640 10 monclient: dump: 2024-04-19T01:46:50.665 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:50.665 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:50.665 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.665 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.665 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:50.665 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:50.665 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:50.665 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:50.665 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:50.665 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:50.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.663+0000 7fa4cf7fe640 10 monclient: _finish_auth 0 2024-04-19T01:46:50.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.663+0000 7fa4cf7fe640 10 monclient: _check_auth_tickets 2024-04-19T01:46:50.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.663+0000 7fa4cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:46:50.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.663+0000 7fa4cf7fe640 1 -- 172.21.15.62:0/4028414772 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4c00ab600 con 0x7fa4d0137780 2024-04-19T01:46:50.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.663+0000 7fa4d7060640 5 monclient: authenticate success, global_id 14802 2024-04-19T01:46:50.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.663+0000 7fa4d7060640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:46:50.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.663+0000 7fa4cf7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:46:50.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.663+0000 7fa4d7060640 10 monclient: get_monmap_and_config success 2024-04-19T01:46:50.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.663+0000 7fa4cf7fe640 1 -- 172.21.15.62:0/4028414772 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c00ab9d0 con 0x7fa4d0137780 2024-04-19T01:46:50.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.663+0000 7fa4d7060640 4 set_mon_vals no callback set 2024-04-19T01:46:50.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.663+0000 7fa4cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:50.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.663+0000 7fa4cf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:46:50.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.663+0000 7fa4cf7fe640 10 monclient: dump: 2024-04-19T01:46:50.667 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:50.667 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:50.667 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.667 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.667 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:50.667 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:50.667 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:50.667 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:50.667 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:50.667 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:50.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d7060640 10 monclient: shutdown 2024-04-19T01:46:50.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d7060640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:46:50.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d7060640 1 -- 172.21.15.62:0/4028414772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d0137780 msgr2=0x7fa4d0137b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:50.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d7060640 1 --2- 172.21.15.62:0/4028414772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d0137780 0x7fa4d0137b60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa4d00a74f0 tx=0x7fa4c001c200 comp rx=0 tx=0).stop 2024-04-19T01:46:50.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d7060640 1 -- 172.21.15.62:0/4028414772 shutdown_connections 2024-04-19T01:46:50.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d7060640 1 --2- 172.21.15.62:0/4028414772 >> v2:172.21.15.71:3300/0 conn(0x7fa4d01380a0 0x7fa4d00a45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:50.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d7060640 1 --2- 172.21.15.62:0/4028414772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d0137780 0x7fa4d0137b60 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:50.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d7060640 1 -- 172.21.15.62:0/4028414772 >> 172.21.15.62:0/4028414772 conn(0x7fa4d00aa1d0 msgr2=0x7fa4d00a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:46:50.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d7060640 1 -- 172.21.15.62:0/4028414772 shutdown_connections 2024-04-19T01:46:50.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d7060640 1 -- 172.21.15.62:0/4028414772 wait complete. 2024-04-19T01:46:50.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d7060640 10 monclient: build_initial_monmap 2024-04-19T01:46:50.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d7060640 10 monclient: monmap: 2024-04-19T01:46:50.668 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:46:50.668 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:46:50.668 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:46:50.668 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:46:50.668 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:46:50.668 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:50.669 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:46:50.669 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:46:50.669 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:46:50.669 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:50.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d7060640 1 Processor -- start 2024-04-19T01:46:50.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d7060640 1 -- start start 2024-04-19T01:46:50.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d7060640 10 monclient: init 2024-04-19T01:46:50.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d7060640 10 monclient: _reopen_session rank -1 2024-04-19T01:46:50.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d7060640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-19T01:46:50.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d7060640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d00a60b0 0x7fa4d01cf610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:50.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d7060640 10 monclient(hunting): picked mon.noname-a con 0x7fa4d00a60b0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:46:50.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d7060640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4d01380a0 0x7fa4d01365d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:50.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d7060640 10 monclient(hunting): picked mon.noname-c con 0x7fa4d01380a0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:46:50.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d7060640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4d0136b10 0x7fa4d013c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:50.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d7060640 10 monclient(hunting): picked mon.noname-b con 0x7fa4d0136b10 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:46:50.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d7060640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:50.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d7060640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d0142170 con 0x7fa4d00a60b0 2024-04-19T01:46:50.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d7060640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:50.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d7060640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d00a5430 con 0x7fa4d0136b10 2024-04-19T01:46:50.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d7060640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:50.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d7060640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fa4d01422f0 con 0x7fa4d01380a0 2024-04-19T01:46:50.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d7060640 10 monclient(hunting): _renew_subs 2024-04-19T01:46:50.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d7060640 10 monclient(hunting): authenticate will time out at 1196.489860s 2024-04-19T01:46:50.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4cffff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4d01380a0 0x7fa4d01365d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:50.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d4dfc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d00a60b0 0x7fa4d01cf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:50.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4cffff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4d01380a0 0x7fa4d01365d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:53626/0 (socket says 172.21.15.62:53626) 2024-04-19T01:46:50.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4cffff640 1 -- 172.21.15.62:0/3623768576 learned_addr learned my addr 172.21.15.62:0/3623768576 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:46:50.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4cffff640 10 monclient(hunting): get_auth_request con 0x7fa4d01380a0 auth_method 0 2024-04-19T01:46:50.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:50.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4cffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:50.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4cffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:50.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d4dfc640 10 monclient(hunting): get_auth_request con 0x7fa4d00a60b0 auth_method 0 2024-04-19T01:46:50.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d4dfc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:50.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d4dfc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:50.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d4dfc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:50.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:50.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:50.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:50.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d4dfc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:50.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d4dfc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:50.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d4dfc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:50.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d55fd640 1 --2- 172.21.15.62:0/3623768576 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4d0136b10 0x7fa4d013c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:50.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d55fd640 10 monclient(hunting): get_auth_request con 0x7fa4d0136b10 auth_method 0 2024-04-19T01:46:50.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d55fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:50.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d55fd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:50.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4d55fd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:50.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4cffff640 10 monclient(hunting): handle_auth_done global_id 5342 payload 824 2024-04-19T01:46:50.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4cffff640 1 -- 172.21.15.62:0/3623768576 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4d0136b10 msgr2=0x7fa4d013c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:50.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4cffff640 1 --2- 172.21.15.62:0/3623768576 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4d0136b10 0x7fa4d013c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:50.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4cffff640 1 -- 172.21.15.62:0/3623768576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d00a60b0 msgr2=0x7fa4d01cf610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:50.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4cffff640 1 --2- 172.21.15.62:0/3623768576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d00a60b0 0x7fa4d01cf610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:50.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4cffff640 10 monclient: _finish_hunting 0 2024-04-19T01:46:50.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4cffff640 1 monclient: found mon.noname-c 2024-04-19T01:46:50.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:46:50.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4cffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:46:50.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.667+0000 7fa4cffff640 1 -- 172.21.15.62:0/3623768576 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4d0120ca0 con 0x7fa4d01380a0 2024-04-19T01:46:50.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4cffff640 1 -- 172.21.15.62:0/3623768576 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4d01380a0 msgr2=0x7fa4d01365d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:50.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4cffff640 1 --2- 172.21.15.62:0/3623768576 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4d01380a0 0x7fa4d01365d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa4c400d800 tx=0x7fa4c400dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:50.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4cd7fa640 1 -- 172.21.15.62:0/3623768576 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c400b870 con 0x7fa4d01380a0 2024-04-19T01:46:50.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:50.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4cd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:46:50.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4cd7fa640 10 monclient: dump: 2024-04-19T01:46:50.678 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:50.678 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:50.678 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.678 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.678 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:50.678 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:50.678 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:50.678 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:50.678 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:50.679 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:50.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4cd7fa640 10 monclient: _finish_auth 0 2024-04-19T01:46:50.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4cd7fa640 10 monclient: _check_auth_tickets 2024-04-19T01:46:50.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:46:50.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4d7060640 5 monclient: authenticate success, global_id 5342 2024-04-19T01:46:50.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4cd7fa640 1 -- 172.21.15.62:0/3623768576 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4c4019070 con 0x7fa4d01380a0 2024-04-19T01:46:50.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4cd7fa640 10 monclient: handle_config config(0 keys) 2024-04-19T01:46:50.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4d7060640 10 monclient: _renew_subs 2024-04-19T01:46:50.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4cd7fa640 1 -- 172.21.15.62:0/3623768576 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c400bc10 con 0x7fa4d01380a0 2024-04-19T01:46:50.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4d7060640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:46:50.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4d7060640 1 -- 172.21.15.62:0/3623768576 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4d0120f80 con 0x7fa4d01380a0 2024-04-19T01:46:50.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:50.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4cd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:46:50.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4ceffd640 4 set_mon_vals no callback set 2024-04-19T01:46:50.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4cd7fa640 10 monclient: dump: 2024-04-19T01:46:50.680 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:50.680 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:50.680 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.680 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.680 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:50.680 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:50.680 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:50.680 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:50.680 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:50.680 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:50.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4d7060640 10 monclient: _renew_subs 2024-04-19T01:46:50.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4d7060640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:46:50.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4d7060640 1 -- 172.21.15.62:0/3623768576 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4d00a2310 con 0x7fa4d01380a0 2024-04-19T01:46:50.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4d7060640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:50.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4d7060640 1 --2- 172.21.15.62:0/3623768576 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4d013ca70 0x7fa4d013ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:50.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4d7060640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:50.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4d7060640 1 -- 172.21.15.62:0/3623768576 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d0120220 con 0x7fa4d013ca70 2024-04-19T01:46:50.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4cd7fa640 1 -- 172.21.15.62:0/3623768576 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fa4c401e460 con 0x7fa4d01380a0 2024-04-19T01:46:50.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4d4dfc640 1 --2- 172.21.15.62:0/3623768576 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4d013ca70 0x7fa4d013ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:50.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4cd7fa640 1 --2- 172.21.15.62:0/3623768576 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa4bc036450 0x7fa4bc038910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:50.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4d4dfc640 10 monclient: get_auth_request con 0x7fa4d013ca70 auth_method 0 2024-04-19T01:46:50.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4d4dfc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:50.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4d4dfc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:50.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4d4dfc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:50.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4cd7fa640 1 -- 172.21.15.62:0/3623768576 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(225..225 src has 108..225) ==== 4346+0+0 (secure 0 0 0) 0x7fa4c40497e0 con 0x7fa4d01380a0 2024-04-19T01:46:50.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4d4dfc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:50.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4d4dfc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:50.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4d4dfc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:50.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4d55fd640 1 --2- 172.21.15.62:0/3623768576 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa4bc036450 0x7fa4bc038910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:50.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4d55fd640 10 monclient: get_auth_request con 0x7fa4bc036450 auth_method 0 2024-04-19T01:46:50.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4d4dfc640 10 monclient(hunting): handle_auth_done global_id 24130 payload 824 2024-04-19T01:46:50.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.671+0000 7fa4d4dfc640 1 -- 172.21.15.62:0/3623768576 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa4d013d390 con 0x7fa4d013ca70 2024-04-19T01:46:50.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.675+0000 7fa4d4dfc640 1 -- 172.21.15.62:0/3623768576 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4d013ca70 msgr2=0x7fa4d013ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:50.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.675+0000 7fa4d4dfc640 1 --2- 172.21.15.62:0/3623768576 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4d013ca70 0x7fa4d013ce50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa4c00ad000 tx=0x7fa4c001aa40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:50.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.675+0000 7fa4d55fd640 1 --2- 172.21.15.62:0/3623768576 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa4bc036450 0x7fa4bc038910 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8007fd0 tx=0x7fa4c8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:50.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.675+0000 7fa4cd7fa640 1 -- 172.21.15.62:0/3623768576 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c00b0050 con 0x7fa4d013ca70 2024-04-19T01:46:50.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.675+0000 7fa4cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:50.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.675+0000 7fa4cd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:46:50.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.675+0000 7fa4cd7fa640 10 monclient: dump: 2024-04-19T01:46:50.682 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:50.682 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:50.682 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.682 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.682 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:50.682 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:50.683 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:50.683 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:50.683 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:50.683 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:50.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.675+0000 7fa4cd7fa640 1 -- 172.21.15.62:0/3623768576 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa4c00b1040 con 0x7fa4d013ca70 2024-04-19T01:46:50.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.675+0000 7fa4cd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:50.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.675+0000 7fa4cd7fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:46:50.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.675+0000 7fa4cd7fa640 1 -- 172.21.15.62:0/3623768576 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4d013ca70 msgr2=0x7fa4d013ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:50.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.675+0000 7fa4cd7fa640 1 --2- 172.21.15.62:0/3623768576 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4d013ca70 0x7fa4d013ce50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa4c00ad000 tx=0x7fa4c001aa40 comp rx=0 tx=0).stop 2024-04-19T01:46:50.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.675+0000 7fa4cd7fa640 1 -- 172.21.15.62:0/3623768576 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4d013ca70 msgr2=0x7fa4d013ce50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:46:50.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.675+0000 7fa4cd7fa640 1 --2- 172.21.15.62:0/3623768576 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4d013ca70 0x7fa4d013ce50 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa4c00ad000 tx=0x7fa4c001aa40 comp rx=0 tx=0).stop 2024-04-19T01:46:50.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.683+0000 7fa4d7060640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:46:50.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.683+0000 7fa4d7060640 1 --2- 172.21.15.62:0/3623768576 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4d01384b0 0x7fa4d0138890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:50.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.683+0000 7fa4d7060640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:50.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.683+0000 7fa4d7060640 1 -- 172.21.15.62:0/3623768576 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d0068ab0 con 0x7fa4d01384b0 2024-04-19T01:46:50.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.683+0000 7fa4d4dfc640 1 --2- 172.21.15.62:0/3623768576 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4d01384b0 0x7fa4d0138890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:50.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.683+0000 7fa4d4dfc640 10 monclient: get_auth_request con 0x7fa4d01384b0 auth_method 0 2024-04-19T01:46:50.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.683+0000 7fa4d4dfc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:50.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.683+0000 7fa4d4dfc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:50.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.683+0000 7fa4d4dfc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:50.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.683+0000 7fa4d4dfc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:50.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.683+0000 7fa4d4dfc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:50.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.683+0000 7fa4d4dfc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:50.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.683+0000 7fa4d4dfc640 10 monclient(hunting): handle_auth_done global_id 24133 payload 824 2024-04-19T01:46:50.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.683+0000 7fa4d4dfc640 1 -- 172.21.15.62:0/3623768576 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa4d00a97b0 con 0x7fa4d01384b0 2024-04-19T01:46:50.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.687+0000 7fa4d4dfc640 1 -- 172.21.15.62:0/3623768576 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4d01384b0 msgr2=0x7fa4d0138890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:50.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.687+0000 7fa4d4dfc640 1 --2- 172.21.15.62:0/3623768576 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4d01384b0 0x7fa4d0138890 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa4c001caa0 tx=0x7fa4c00a8000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:50.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.687+0000 7fa4cd7fa640 1 -- 172.21.15.62:0/3623768576 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c00bfb30 con 0x7fa4d01384b0 2024-04-19T01:46:50.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.687+0000 7fa4cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:50.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.687+0000 7fa4cd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:46:50.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.687+0000 7fa4cd7fa640 10 monclient: dump: 2024-04-19T01:46:50.687 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:50.687 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:50.687 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.687 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.687 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:50.687 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:50.687 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:50.687 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:50.687 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:50.687 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:50.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.687+0000 7fa4cd7fa640 1 -- 172.21.15.62:0/3623768576 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7fa4c00bfb30 con 0x7fa4d01384b0 2024-04-19T01:46:50.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.687+0000 7fa4cd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:46:50.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.687+0000 7fa4cd7fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:46:50.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.687+0000 7fa4cd7fa640 1 -- 172.21.15.62:0/3623768576 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4d01384b0 msgr2=0x7fa4d0138890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:50.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.687+0000 7fa4cd7fa640 1 --2- 172.21.15.62:0/3623768576 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4d01384b0 0x7fa4d0138890 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa4c001caa0 tx=0x7fa4c00a8000 comp rx=0 tx=0).stop 2024-04-19T01:46:50.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.687+0000 7fa4cd7fa640 1 -- 172.21.15.62:0/3623768576 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4d01384b0 msgr2=0x7fa4d0138890 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:46:50.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.687+0000 7fa4cd7fa640 1 --2- 172.21.15.62:0/3623768576 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4d01384b0 0x7fa4d0138890 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa4c001caa0 tx=0x7fa4c00a8000 comp rx=0 tx=0).stop 2024-04-19T01:46:50.688 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-19T01:46:50.688 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "b", 2024-04-19T01:46:50.689 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 1, 2024-04-19T01:46:50.689 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "peon", 2024-04-19T01:46:50.689 INFO:teuthology.orchestra.run.smithi062.stdout: "election_epoch": 34, 2024-04-19T01:46:50.689 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 17335, 2024-04-19T01:46:50.689 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum": [ 2024-04-19T01:46:50.689 INFO:teuthology.orchestra.run.smithi062.stdout: 0, 2024-04-19T01:46:50.689 INFO:teuthology.orchestra.run.smithi062.stdout: 1, 2024-04-19T01:46:50.689 INFO:teuthology.orchestra.run.smithi062.stdout: 2 2024-04-19T01:46:50.689 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:46:50.689 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_age": 7, 2024-04-19T01:46:50.689 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:46:50.689 INFO:teuthology.orchestra.run.smithi062.stdout: "required_con": "2450521708007866372", 2024-04-19T01:46:50.689 INFO:teuthology.orchestra.run.smithi062.stdout: "required_mon": [ 2024-04-19T01:46:50.689 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:46:50.689 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:46:50.690 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:46:50.690 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:46:50.690 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:46:50.690 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:46:50.690 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:46:50.690 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:46:50.690 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:46:50.690 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:46:50.690 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:46:50.690 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:46:50.690 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:46:50.690 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_mon": [ 2024-04-19T01:46:50.690 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:46:50.690 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:46:50.691 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:46:50.691 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:46:50.691 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:46:50.691 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:46:50.691 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:46:50.691 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:46:50.691 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:46:50.691 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:46:50.691 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:46:50.691 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:50.691 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.691 INFO:teuthology.orchestra.run.smithi062.stdout: "outside_quorum": [], 2024-04-19T01:46:50.691 INFO:teuthology.orchestra.run.smithi062.stdout: "extra_probe_peers": [], 2024-04-19T01:46:50.691 INFO:teuthology.orchestra.run.smithi062.stdout: "sync_provider": [], 2024-04-19T01:46:50.691 INFO:teuthology.orchestra.run.smithi062.stdout: "monmap": { 2024-04-19T01:46:50.692 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 1, 2024-04-19T01:46:50.692 INFO:teuthology.orchestra.run.smithi062.stdout: "fsid": "6466e4b6-2520-485f-95b3-9401915fbbad", 2024-04-19T01:46:50.692 INFO:teuthology.orchestra.run.smithi062.stdout: "modified": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:46:50.692 INFO:teuthology.orchestra.run.smithi062.stdout: "created": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:46:50.692 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release": 19, 2024-04-19T01:46:50.692 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release_name": "squid", 2024-04-19T01:46:50.692 INFO:teuthology.orchestra.run.smithi062.stdout: "election_strategy": 1, 2024-04-19T01:46:50.692 INFO:teuthology.orchestra.run.smithi062.stdout: "disallowed_leaders: ": "", 2024-04-19T01:46:50.692 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false, 2024-04-19T01:46:50.692 INFO:teuthology.orchestra.run.smithi062.stdout: "tiebreaker_mon": "", 2024-04-19T01:46:50.692 INFO:teuthology.orchestra.run.smithi062.stdout: "removed_ranks: ": "", 2024-04-19T01:46:50.692 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:46:50.692 INFO:teuthology.orchestra.run.smithi062.stdout: "persistent": [ 2024-04-19T01:46:50.692 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:46:50.692 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:46:50.692 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:46:50.693 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:46:50.693 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:46:50.693 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:46:50.693 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:46:50.693 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:46:50.693 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:46:50.693 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:46:50.693 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:46:50.693 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:46:50.693 INFO:teuthology.orchestra.run.smithi062.stdout: "optional": [] 2024-04-19T01:46:50.693 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.693 INFO:teuthology.orchestra.run.smithi062.stdout: "mons": [ 2024-04-19T01:46:50.693 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.693 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 0, 2024-04-19T01:46:50.693 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "a", 2024-04-19T01:46:50.694 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:46:50.694 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:46:50.694 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.694 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:46:50.694 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3300", 2024-04-19T01:46:50.694 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:50.694 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.694 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.694 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:46:50.694 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789", 2024-04-19T01:46:50.694 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:50.694 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:50.694 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:50.694 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.695 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789/0", 2024-04-19T01:46:50.695 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6789/0", 2024-04-19T01:46:50.695 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:46:50.695 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:46:50.695 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:46:50.695 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.695 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.695 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 1, 2024-04-19T01:46:50.695 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "b", 2024-04-19T01:46:50.695 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:46:50.695 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:46:50.695 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.695 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:46:50.695 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:3300", 2024-04-19T01:46:50.695 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:50.696 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.696 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.696 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:46:50.696 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789", 2024-04-19T01:46:50.696 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:50.696 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:50.696 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:50.696 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.696 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789/0", 2024-04-19T01:46:50.696 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.71:6789/0", 2024-04-19T01:46:50.696 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:46:50.696 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:46:50.696 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:46:50.696 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.696 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.696 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 2, 2024-04-19T01:46:50.697 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "c", 2024-04-19T01:46:50.697 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:46:50.697 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:46:50.697 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.697 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:46:50.697 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3301", 2024-04-19T01:46:50.697 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:50.697 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.697 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.697 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:46:50.697 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790", 2024-04-19T01:46:50.697 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:46:50.697 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:50.697 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:50.697 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.698 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790/0", 2024-04-19T01:46:50.698 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6790/0", 2024-04-19T01:46:50.698 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:46:50.698 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:46:50.698 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:46:50.698 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:50.698 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:50.698 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.698 INFO:teuthology.orchestra.run.smithi062.stdout: "feature_map": { 2024-04-19T01:46:50.698 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": [ 2024-04-19T01:46:50.698 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.698 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:46:50.698 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:46:50.698 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:46:50.699 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:50.699 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:46:50.699 INFO:teuthology.orchestra.run.smithi062.stdout: "client": [ 2024-04-19T01:46:50.699 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:46:50.699 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:46:50.699 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:46:50.699 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 3 2024-04-19T01:46:50.699 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:46:50.699 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:46:50.699 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:46:50.699 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false 2024-04-19T01:46:50.699 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-19T01:46:50.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.687+0000 7fa4d7060640 1 -- 172.21.15.62:0/3623768576 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa4bc036450 msgr2=0x7fa4bc038910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:50.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.687+0000 7fa4d7060640 1 --2- 172.21.15.62:0/3623768576 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa4bc036450 0x7fa4bc038910 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8007fd0 tx=0x7fa4c8009040 comp rx=0 tx=0).stop 2024-04-19T01:46:50.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.687+0000 7fa4d7060640 10 monclient: shutdown 2024-04-19T01:46:50.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.687+0000 7fa4d7060640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:46:50.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.687+0000 7fa4d7060640 1 -- 172.21.15.62:0/3623768576 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4d01380a0 msgr2=0x7fa4d01365d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:50.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.687+0000 7fa4d7060640 1 --2- 172.21.15.62:0/3623768576 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa4d01380a0 0x7fa4d01365d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa4c400d800 tx=0x7fa4c400dcc0 comp rx=0 tx=0).stop 2024-04-19T01:46:50.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.687+0000 7fa4d55fd640 1 -- 172.21.15.62:0/3623768576 reap_dead start 2024-04-19T01:46:50.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.687+0000 7fa4d55fd640 1 -- 172.21.15.62:0/3623768576 reap_dead start 2024-04-19T01:46:50.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.687+0000 7fa4d7060640 1 -- 172.21.15.62:0/3623768576 shutdown_connections 2024-04-19T01:46:50.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.687+0000 7fa4d7060640 1 -- 172.21.15.62:0/3623768576 >> 172.21.15.62:0/3623768576 conn(0x7fa4d00aa1d0 msgr2=0x7fa4d00a79d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:46:50.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.687+0000 7fa4d7060640 1 -- 172.21.15.62:0/3623768576 shutdown_connections 2024-04-19T01:46:50.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.687+0000 7fa4d7060640 1 -- 172.21.15.62:0/3623768576 wait complete. 2024-04-19T01:46:50.702 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-19T01:46:50.702 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-04-19T01:46:50.702 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-19T01:46:50.702 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-04-19T01:46:50.702 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-19T01:46:50.703 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-19T01:46:50.703 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:46:50.716 INFO:tasks.ceph.mon.c:Stopped 2024-04-19T01:46:50.716 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-19T01:46:50.716 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:46:50.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4caec99640 10 monclient: get_monmap_and_config 2024-04-19T01:46:50.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4caec99640 10 monclient: build_initial_monmap 2024-04-19T01:46:50.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4caec99640 10 monclient: monmap: 2024-04-19T01:46:50.873 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:46:50.873 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:50.873 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:46:50.879132+0000 2024-04-19T01:46:50.873 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:46:50.879132+0000 2024-04-19T01:46:50.874 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:46:50.874 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:50.874 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:46:50.874 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:46:50.874 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:46:50.874 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:46:50.874 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:46:50.874 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:50.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4caec99640 1 Processor -- start 2024-04-19T01:46:50.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4caec99640 1 -- start start 2024-04-19T01:46:50.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4caec99640 10 monclient: init 2024-04-19T01:46:50.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4caec99640 10 monclient: _reopen_session rank -1 2024-04-19T01:46:50.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4caec99640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-04-19T01:46:50.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4caec99640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4ca80a2460 0x7f4ca80a2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:50.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4caec99640 10 monclient(hunting): picked mon.noname-c con 0x7f4ca80a2460 addr v2:172.21.15.71:3300/0 2024-04-19T01:46:50.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4caec99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4ca8075fe0 addr v1:172.21.15.71:6789/0 2024-04-19T01:46:50.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4caec99640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4ca80a2d80 0x7f4ca8134920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:50.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4caec99640 10 monclient(hunting): picked mon.noname-a con 0x7f4ca80a2d80 addr v2:172.21.15.62:3300/0 2024-04-19T01:46:50.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4caec99640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ca809ff10 con 0x7f4ca8075fe0 2024-04-19T01:46:50.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4caec99640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:50.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4caec99640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f4ca80a0840 con 0x7f4ca80a2d80 2024-04-19T01:46:50.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4caec99640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:50.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4caec99640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f4ca80a0b20 con 0x7f4ca80a2460 2024-04-19T01:46:50.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4caec99640 10 monclient(hunting): _renew_subs 2024-04-19T01:46:50.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4caec99640 10 monclient(hunting): authenticate will time out at 1196.698643s 2024-04-19T01:46:50.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4cad236640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4ca80a2d80 0x7f4ca8134920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:50.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4cad236640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4ca80a2d80 0x7f4ca8134920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44294/0 (socket says 172.21.15.62:44294) 2024-04-19T01:46:50.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4cad236640 1 -- 172.21.15.62:0/878465622 learned_addr learned my addr 172.21.15.62:0/878465622 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:46:50.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4cad236640 10 monclient(hunting): get_auth_request con 0x7f4ca80a2d80 auth_method 0 2024-04-19T01:46:50.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4cad236640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:50.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4cad236640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:50.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4cad236640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:50.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4cad236640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:50.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4cad236640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:50.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4cad236640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:50.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4caca35640 1 --2- 172.21.15.62:0/878465622 >> v2:172.21.15.71:3300/0 conn(0x7f4ca80a2460 0x7f4ca80a2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:50.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4cad236640 10 monclient(hunting): handle_auth_done global_id 14808 payload 274 2024-04-19T01:46:50.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4cad236640 1 -- 172.21.15.62:0/878465622 >> v2:172.21.15.71:3300/0 conn(0x7f4ca80a2460 msgr2=0x7f4ca80a2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:46:50.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4cad236640 1 --2- 172.21.15.62:0/878465622 >> v2:172.21.15.71:3300/0 conn(0x7f4ca80a2460 0x7f4ca80a2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:50.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4cad236640 1 -- 172.21.15.62:0/878465622 >> v1:172.21.15.71:6789/0 conn(0x7f4ca8075fe0 legacy=0x7f4ca8134210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:46:50.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4cad236640 10 monclient: _finish_hunting 0 2024-04-19T01:46:50.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4cad236640 1 monclient: found mon.noname-a 2024-04-19T01:46:50.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4cad236640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:46:50.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4cad236640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:46:50.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4cad236640 1 -- 172.21.15.62:0/878465622 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ca80a0140 con 0x7f4ca80a2d80 2024-04-19T01:46:50.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4caca35640 10 monclient: get_auth_request con 0x7f4ca80a2460 auth_method 0 2024-04-19T01:46:50.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.875+0000 7f4caca35640 1 --2- 172.21.15.62:0/878465622 >> v2:172.21.15.71:3300/0 conn(0x7f4ca80a2460 0x7f4ca80a2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:46:50.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.879+0000 7f4cad236640 1 -- 172.21.15.62:0/878465622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ca80a2d80 msgr2=0x7f4ca8134920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:50.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.879+0000 7f4cad236640 1 --2- 172.21.15.62:0/878465622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ca80a2d80 0x7f4ca8134920 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4ca80a7a00 tx=0x7f4ca001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f43e4a0579e602c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:50.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.879+0000 7f4ca77fe640 1 -- 172.21.15.62:0/878465622 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca00b0070 con 0x7f4ca80a2d80 2024-04-19T01:46:50.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.879+0000 7f4ca77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:50.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.879+0000 7f4ca77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:46:50.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.879+0000 7f4ca77fe640 10 monclient: dump: 2024-04-19T01:46:50.886 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:50.886 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:50.886 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.886 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.886 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:50.886 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:50.886 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:50.886 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:50.886 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:50.886 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:50.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.879+0000 7f4ca77fe640 10 monclient: _finish_auth 0 2024-04-19T01:46:50.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.879+0000 7f4ca77fe640 10 monclient: _check_auth_tickets 2024-04-19T01:46:50.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.879+0000 7f4ca77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:46:50.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.879+0000 7f4ca77fe640 1 -- 172.21.15.62:0/878465622 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ca00ab600 con 0x7f4ca80a2d80 2024-04-19T01:46:50.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.879+0000 7f4ca77fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:46:50.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.879+0000 7f4ca77fe640 1 -- 172.21.15.62:0/878465622 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca00ab9d0 con 0x7f4ca80a2d80 2024-04-19T01:46:50.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.879+0000 7f4caec99640 5 monclient: authenticate success, global_id 14808 2024-04-19T01:46:50.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.879+0000 7f4caec99640 10 monclient: get_monmap_and_config success 2024-04-19T01:46:50.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.879+0000 7f4caec99640 4 set_mon_vals no callback set 2024-04-19T01:46:50.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.879+0000 7f4ca77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:50.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.879+0000 7f4ca77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:46:50.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.879+0000 7f4ca77fe640 10 monclient: dump: 2024-04-19T01:46:50.887 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:50.887 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:50.887 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.887 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.887 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:50.887 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:50.888 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:50.888 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:50.888 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:50.888 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:50.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caec99640 10 monclient: shutdown 2024-04-19T01:46:50.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caec99640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:46:50.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caec99640 1 -- 172.21.15.62:0/878465622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ca80a2d80 msgr2=0x7f4ca8134920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:50.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caec99640 1 --2- 172.21.15.62:0/878465622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ca80a2d80 0x7f4ca8134920 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4ca80a7a00 tx=0x7f4ca001c3e0 comp rx=0 tx=0).stop 2024-04-19T01:46:50.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caec99640 1 -- 172.21.15.62:0/878465622 shutdown_connections 2024-04-19T01:46:50.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caec99640 1 --2- 172.21.15.62:0/878465622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ca80a2d80 0x7f4ca8134920 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:50.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caec99640 1 --2- 172.21.15.62:0/878465622 >> v2:172.21.15.71:3300/0 conn(0x7f4ca80a2460 0x7f4ca80a2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:50.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caec99640 1 -- 172.21.15.62:0/878465622 >> 172.21.15.62:0/878465622 conn(0x7f4ca80a7050 msgr2=0x7f4ca80a3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:46:50.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caec99640 1 -- 172.21.15.62:0/878465622 shutdown_connections 2024-04-19T01:46:50.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caec99640 1 -- 172.21.15.62:0/878465622 wait complete. 2024-04-19T01:46:50.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caec99640 10 monclient: build_initial_monmap 2024-04-19T01:46:50.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caec99640 10 monclient: monmap: 2024-04-19T01:46:50.889 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:46:50.889 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:46:50.889 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:46:50.889 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:46:50.889 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:46:50.889 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:50.889 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:46:50.889 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:46:50.889 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:46:50.889 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:50.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caec99640 1 Processor -- start 2024-04-19T01:46:50.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caec99640 1 -- start start 2024-04-19T01:46:50.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caec99640 10 monclient: init 2024-04-19T01:46:50.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caec99640 10 monclient: _reopen_session rank -1 2024-04-19T01:46:50.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caec99640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-19T01:46:50.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caec99640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ca8075fe0 0x7f4ca812d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:50.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caec99640 10 monclient(hunting): picked mon.noname-a con 0x7f4ca8075fe0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:46:50.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caec99640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4ca80a2460 0x7f4ca8135c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:50.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caec99640 10 monclient(hunting): picked mon.noname-c con 0x7f4ca80a2460 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:46:50.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caec99640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ca80a2d80 0x7f4ca8136170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:50.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caec99640 10 monclient(hunting): picked mon.noname-b con 0x7f4ca80a2d80 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:46:50.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caec99640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:50.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caec99640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca8134fe0 con 0x7f4ca8075fe0 2024-04-19T01:46:50.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caec99640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:50.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caec99640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca8134e60 con 0x7f4ca80a2d80 2024-04-19T01:46:50.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caec99640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:50.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caec99640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f4ca8142190 con 0x7f4ca80a2460 2024-04-19T01:46:50.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caec99640 10 monclient(hunting): _renew_subs 2024-04-19T01:46:50.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caec99640 10 monclient(hunting): authenticate will time out at 1196.706072s 2024-04-19T01:46:50.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4ca7fff640 1 -- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4ca80a2460 msgr2=0x7f4ca8135c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:3301/0 2024-04-19T01:46:50.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4ca7fff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4ca80a2460 0x7f4ca8135c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:46:50.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caca35640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ca8075fe0 0x7f4ca812d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:50.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caca35640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ca8075fe0 0x7f4ca812d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44310/0 (socket says 172.21.15.62:44310) 2024-04-19T01:46:50.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caca35640 1 -- 172.21.15.62:0/1576388923 learned_addr learned my addr 172.21.15.62:0/1576388923 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:46:50.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caca35640 10 monclient(hunting): get_auth_request con 0x7f4ca8075fe0 auth_method 0 2024-04-19T01:46:50.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caca35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:50.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caca35640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:50.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caca35640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:50.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caca35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:50.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caca35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:50.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caca35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:50.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4cad236640 1 --2- 172.21.15.62:0/1576388923 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ca80a2d80 0x7f4ca8136170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:50.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caca35640 10 monclient(hunting): handle_auth_done global_id 14811 payload 824 2024-04-19T01:46:50.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caca35640 1 -- 172.21.15.62:0/1576388923 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4ca80a2460 msgr2=0x7f4ca8135c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:46:50.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caca35640 1 --2- 172.21.15.62:0/1576388923 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4ca80a2460 0x7f4ca8135c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:50.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caca35640 1 -- 172.21.15.62:0/1576388923 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ca80a2d80 msgr2=0x7f4ca8136170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:50.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caca35640 1 --2- 172.21.15.62:0/1576388923 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ca80a2d80 0x7f4ca8136170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:50.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caca35640 10 monclient: _finish_hunting 0 2024-04-19T01:46:50.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caca35640 1 monclient: found mon.noname-a 2024-04-19T01:46:50.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caca35640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:46:50.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caca35640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:46:50.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4caca35640 1 -- 172.21.15.62:0/1576388923 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ca8139b70 con 0x7f4ca8075fe0 2024-04-19T01:46:50.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4cad236640 10 monclient: get_auth_request con 0x7f4ca80a2d80 auth_method 0 2024-04-19T01:46:50.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.883+0000 7f4cad236640 1 --2- 172.21.15.62:0/1576388923 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ca80a2d80 0x7f4ca8136170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:46:50.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.887+0000 7f4caca35640 1 -- 172.21.15.62:0/1576388923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ca8075fe0 msgr2=0x7f4ca812d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:50.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.887+0000 7f4caca35640 1 --2- 172.21.15.62:0/1576388923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ca8075fe0 0x7f4ca812d7c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4c9800cf60 tx=0x7f4c9800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:50.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.887+0000 7f4ca57fa640 1 -- 172.21.15.62:0/1576388923 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9801a070 con 0x7f4ca8075fe0 2024-04-19T01:46:50.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.887+0000 7f4ca57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:50.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.887+0000 7f4ca57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:46:50.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.887+0000 7f4ca57fa640 10 monclient: dump: 2024-04-19T01:46:50.893 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:50.893 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:50.893 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.893 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.893 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:50.893 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:50.893 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:50.893 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:50.893 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:50.893 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:50.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.887+0000 7f4ca57fa640 10 monclient: _finish_auth 0 2024-04-19T01:46:50.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.887+0000 7f4ca57fa640 10 monclient: _check_auth_tickets 2024-04-19T01:46:50.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.887+0000 7f4ca57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:46:50.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.887+0000 7f4ca57fa640 1 -- 172.21.15.62:0/1576388923 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c9800da50 con 0x7f4ca8075fe0 2024-04-19T01:46:50.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.887+0000 7f4ca57fa640 10 monclient: handle_config config(0 keys) 2024-04-19T01:46:50.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.887+0000 7f4ca57fa640 1 -- 172.21.15.62:0/1576388923 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9800de20 con 0x7f4ca8075fe0 2024-04-19T01:46:50.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.887+0000 7f4caec99640 5 monclient: authenticate success, global_id 14811 2024-04-19T01:46:50.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.887+0000 7f4caec99640 10 monclient: _renew_subs 2024-04-19T01:46:50.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.887+0000 7f4caec99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:46:50.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.887+0000 7f4caec99640 1 -- 172.21.15.62:0/1576388923 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ca8137880 con 0x7f4ca8075fe0 2024-04-19T01:46:50.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.887+0000 7f4ca57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:50.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.887+0000 7f4ca57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:46:50.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.887+0000 7f4ca57fa640 10 monclient: dump: 2024-04-19T01:46:50.894 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:50.894 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:50.894 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.894 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:50.894 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:50.895 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:50.895 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:50.895 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:50.895 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:50.895 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:50.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.887+0000 7f4ca67fc640 4 set_mon_vals no callback set 2024-04-19T01:46:50.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.887+0000 7f4caec99640 10 monclient: _renew_subs 2024-04-19T01:46:50.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.887+0000 7f4caec99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:46:50.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.887+0000 7f4caec99640 1 -- 172.21.15.62:0/1576388923 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ca81395b0 con 0x7f4ca8075fe0 2024-04-19T01:46:50.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.887+0000 7f4caec99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:50.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.887+0000 7f4caec99640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:50.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.887+0000 7f4caec99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:46:50.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.887+0000 7f4caec99640 1 -- 172.21.15.62:0/1576388923 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ca81397b0 con 0x7f4ca8075fe0 2024-04-19T01:46:50.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.887+0000 7f4ca57fa640 1 -- 172.21.15.62:0/1576388923 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f4c9801d070 con 0x7f4ca8075fe0 2024-04-19T01:46:50.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.887+0000 7f4ca57fa640 1 --2- 172.21.15.62:0/1576388923 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4c94036450 0x7f4c94038910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:50.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.887+0000 7f4ca57fa640 1 -- 172.21.15.62:0/1576388923 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(225..225 src has 108..225) ==== 4346+0+0 (secure 0 0 0) 0x7f4c98049c70 con 0x7f4ca8075fe0 2024-04-19T01:46:50.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.887+0000 7f4ca7fff640 1 --2- 172.21.15.62:0/1576388923 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4c94036450 0x7f4c94038910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:50.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.887+0000 7f4ca7fff640 10 monclient: get_auth_request con 0x7f4c94036450 auth_method 0 2024-04-19T01:46:50.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.891+0000 7f4ca7fff640 1 --2- 172.21.15.62:0/1576388923 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4c94036450 0x7f4c94038910 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f4ca8137480 tx=0x7f4c9c007a20 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:50.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.891+0000 7f4ca57fa640 1 -- 172.21.15.62:0/1576388923 DispatchQueue::entry inject delay of 0.002000 2024-04-19T01:46:50.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.891+0000 7f4ca57fa640 1 -- 172.21.15.62:0/1576388923 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4ca81397b0 con 0x7f4ca8075fe0 2024-04-19T01:46:50.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.891+0000 7f4ca57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:50.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:50.891+0000 7f4ca57fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:46:51.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:51.119+0000 7f4caec99640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:46:51.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:51.119+0000 7f4caec99640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:46:51.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:51.119+0000 7f4caec99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:46:51.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:51.119+0000 7f4caec99640 1 -- 172.21.15.62:0/1576388923 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4ca8068ab0 con 0x7f4ca8075fe0 2024-04-19T01:46:51.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:51.123+0000 7f4ca57fa640 1 -- 172.21.15.62:0/1576388923 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f4c98013340 con 0x7f4ca8075fe0 2024-04-19T01:46:51.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:51.123+0000 7f4ca57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:46:51.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:51.123+0000 7f4ca57fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:46:51.121 INFO:teuthology.orchestra.run.smithi062.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","modified":"2024-04-19T01:41:38.497795Z","created":"2024-04-19T01:41:38.497795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:46:51.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:51.123+0000 7f4caec99640 1 -- 172.21.15.62:0/1576388923 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4c94036450 msgr2=0x7f4c94038910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:51.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:51.123+0000 7f4caec99640 1 --2- 172.21.15.62:0/1576388923 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4c94036450 0x7f4c94038910 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f4ca8137480 tx=0x7f4c9c007a20 comp rx=0 tx=0).stop 2024-04-19T01:46:51.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:51.123+0000 7f4caec99640 10 monclient: shutdown 2024-04-19T01:46:51.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:51.123+0000 7f4caec99640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:46:51.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:51.123+0000 7f4caec99640 1 -- 172.21.15.62:0/1576388923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ca8075fe0 msgr2=0x7f4ca812d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:51.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:51.123+0000 7f4caec99640 1 --2- 172.21.15.62:0/1576388923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ca8075fe0 0x7f4ca812d7c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4c9800cf60 tx=0x7f4c9800cf90 comp rx=0 tx=0).stop 2024-04-19T01:46:51.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:51.123+0000 7f4caec99640 1 -- 172.21.15.62:0/1576388923 shutdown_connections 2024-04-19T01:46:51.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:51.123+0000 7f4caec99640 1 --2- 172.21.15.62:0/1576388923 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4c94036450 0x7f4c94038910 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:51.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:51.123+0000 7f4caec99640 1 --2- 172.21.15.62:0/1576388923 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4ca80a2460 0x7f4ca8135c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:51.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:51.123+0000 7f4caec99640 1 --2- 172.21.15.62:0/1576388923 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ca80a2d80 0x7f4ca8136170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:51.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:51.123+0000 7f4caec99640 1 --2- 172.21.15.62:0/1576388923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ca8075fe0 0x7f4ca812d7c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:51.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:51.123+0000 7f4caec99640 1 -- 172.21.15.62:0/1576388923 >> 172.21.15.62:0/1576388923 conn(0x7f4ca80a7050 msgr2=0x7f4ca8071b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:46:51.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:51.123+0000 7f4caec99640 1 -- 172.21.15.62:0/1576388923 shutdown_connections 2024-04-19T01:46:51.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:51.123+0000 7f4caec99640 1 -- 172.21.15.62:0/1576388923 wait complete. 2024-04-19T01:46:54.141 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:46:54.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.275+0000 7f6c9ca1f640 10 monclient: get_monmap_and_config 2024-04-19T01:46:54.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.275+0000 7f6c9ca1f640 10 monclient: build_initial_monmap 2024-04-19T01:46:54.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.275+0000 7f6c9ca1f640 10 monclient: monmap: 2024-04-19T01:46:54.274 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:46:54.277 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:54.277 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:46:54.279880+0000 2024-04-19T01:46:54.277 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:46:54.279880+0000 2024-04-19T01:46:54.278 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:46:54.278 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:54.278 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:46:54.278 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:46:54.278 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:46:54.278 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:46:54.278 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:46:54.278 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:54.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.275+0000 7f6c9ca1f640 1 Processor -- start 2024-04-19T01:46:54.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.275+0000 7f6c9ca1f640 1 -- start start 2024-04-19T01:46:54.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.275+0000 7f6c9ca1f640 10 monclient: init 2024-04-19T01:46:54.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.275+0000 7f6c9ca1f640 10 monclient: _reopen_session rank -1 2024-04-19T01:46:54.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.275+0000 7f6c9ca1f640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-04-19T01:46:54.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.275+0000 7f6c9ca1f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6c980a47e0 0x7f6c980a4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:54.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.275+0000 7f6c9ca1f640 10 monclient(hunting): picked mon.noname-a con 0x7f6c980a47e0 addr v2:172.21.15.62:3300/0 2024-04-19T01:46:54.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.275+0000 7f6c9ca1f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6c980a5100 addr v1:172.21.15.62:6789/0 2024-04-19T01:46:54.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.275+0000 7f6c9ca1f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c981357c0 0x7f6c9813d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:54.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.275+0000 7f6c9ca1f640 10 monclient(hunting): picked mon.noname-b con 0x7f6c981357c0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:46:54.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.275+0000 7f6c9ca1f640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c9809f7c0 con 0x7f6c980a5100 2024-04-19T01:46:54.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.275+0000 7f6c9ca1f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:54.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.275+0000 7f6c9ca1f640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f6c9809bf00 con 0x7f6c980a47e0 2024-04-19T01:46:54.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.275+0000 7f6c9ca1f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:54.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.275+0000 7f6c9ca1f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f6c9809ee90 con 0x7f6c981357c0 2024-04-19T01:46:54.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.275+0000 7f6c9ca1f640 10 monclient(hunting): _renew_subs 2024-04-19T01:46:54.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.275+0000 7f6c9ca1f640 10 monclient(hunting): authenticate will time out at 1200.099725s 2024-04-19T01:46:54.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.275+0000 7f6c96d9d640 1 -- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c981357c0 msgr2=0x7f6c9813d600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.62:3301/0 2024-04-19T01:46:54.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.275+0000 7f6c96d9d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c981357c0 0x7f6c9813d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:46:54.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.275+0000 7f6c9659c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6c980a47e0 0x7f6c980a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:54.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.275+0000 7f6c95d9b640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f6c980a5100 0x7f6c981350b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:39844/0 (socket says 172.21.15.62:39844) 2024-04-19T01:46:54.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.275+0000 7f6c95d9b640 1 -- 172.21.15.62:0/307459414 learned_addr learned my addr 172.21.15.62:0/307459414 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:46:54.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.275+0000 7f6c9659c640 10 monclient(hunting): get_auth_request con 0x7f6c980a47e0 auth_method 0 2024-04-19T01:46:54.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.275+0000 7f6c9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:54.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.275+0000 7f6c9659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:54.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.275+0000 7f6c9659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:54.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.279+0000 7f6c9659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:54.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.279+0000 7f6c9659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:54.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.279+0000 7f6c9659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:54.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.279+0000 7f6c9659c640 10 monclient(hunting): handle_auth_done global_id 14814 payload 274 2024-04-19T01:46:54.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.279+0000 7f6c9659c640 1 -- 172.21.15.62:0/307459414 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c981357c0 msgr2=0x7f6c9813d600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:46:54.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.279+0000 7f6c9659c640 1 --2- 172.21.15.62:0/307459414 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c981357c0 0x7f6c9813d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:54.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.279+0000 7f6c9659c640 1 -- 172.21.15.62:0/307459414 >> v1:172.21.15.62:6789/0 conn(0x7f6c980a5100 legacy=0x7f6c981350b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:46:54.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.279+0000 7f6c9659c640 10 monclient: _finish_hunting 0 2024-04-19T01:46:54.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.279+0000 7f6c9659c640 1 monclient: found mon.noname-a 2024-04-19T01:46:54.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.279+0000 7f6c9659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:46:54.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.279+0000 7f6c9659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:46:54.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.279+0000 7f6c9659c640 1 -- 172.21.15.62:0/307459414 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c9809c2d0 con 0x7f6c980a47e0 2024-04-19T01:46:54.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9659c640 1 -- 172.21.15.62:0/307459414 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c980a47e0 msgr2=0x7f6c980a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:54.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9659c640 1 --2- 172.21.15.62:0/307459414 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c980a47e0 0x7f6c980a4bc0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6c880097f0 tx=0x7f6c8801c230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d46da79d047b34c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:54.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9559a640 1 -- 172.21.15.62:0/307459414 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c880b0070 con 0x7f6c980a47e0 2024-04-19T01:46:54.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:54.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:46:54.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9559a640 10 monclient: dump: 2024-04-19T01:46:54.285 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:54.285 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:54.285 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:54.285 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:54.285 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:54.285 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:54.285 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:54.285 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:54.285 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:54.285 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:54.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9559a640 10 monclient: _finish_auth 0 2024-04-19T01:46:54.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9559a640 10 monclient: _check_auth_tickets 2024-04-19T01:46:54.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:46:54.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 5 monclient: authenticate success, global_id 14814 2024-04-19T01:46:54.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:46:54.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9559a640 1 -- 172.21.15.62:0/307459414 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c880ab600 con 0x7f6c980a47e0 2024-04-19T01:46:54.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9559a640 10 monclient: handle_config config(0 keys) 2024-04-19T01:46:54.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 10 monclient: get_monmap_and_config success 2024-04-19T01:46:54.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 4 set_mon_vals no callback set 2024-04-19T01:46:54.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9559a640 1 -- 172.21.15.62:0/307459414 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c880ab9d0 con 0x7f6c980a47e0 2024-04-19T01:46:54.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:54.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:46:54.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9559a640 10 monclient: dump: 2024-04-19T01:46:54.286 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:54.286 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:54.286 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:54.286 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:54.286 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:54.286 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:54.287 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:54.287 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:54.287 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:54.287 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:54.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 10 monclient: shutdown 2024-04-19T01:46:54.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:46:54.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 1 -- 172.21.15.62:0/307459414 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c980a47e0 msgr2=0x7f6c980a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:54.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 1 --2- 172.21.15.62:0/307459414 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c980a47e0 0x7f6c980a4bc0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6c880097f0 tx=0x7f6c8801c230 comp rx=0 tx=0).stop 2024-04-19T01:46:54.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 1 -- 172.21.15.62:0/307459414 shutdown_connections 2024-04-19T01:46:54.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 1 --2- 172.21.15.62:0/307459414 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c981357c0 0x7f6c9813d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:54.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 1 --2- 172.21.15.62:0/307459414 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c980a47e0 0x7f6c980a4bc0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:54.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 1 -- 172.21.15.62:0/307459414 >> 172.21.15.62:0/307459414 conn(0x7f6c980a23d0 msgr2=0x7f6c980a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:46:54.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 1 -- 172.21.15.62:0/307459414 shutdown_connections 2024-04-19T01:46:54.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 1 -- 172.21.15.62:0/307459414 wait complete. 2024-04-19T01:46:54.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 10 monclient: build_initial_monmap 2024-04-19T01:46:54.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 10 monclient: monmap: 2024-04-19T01:46:54.291 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:46:54.292 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:46:54.292 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:46:54.292 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:46:54.292 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:46:54.292 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:54.292 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:46:54.292 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:46:54.292 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:46:54.292 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:54.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 1 Processor -- start 2024-04-19T01:46:54.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 1 -- start start 2024-04-19T01:46:54.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 10 monclient: init 2024-04-19T01:46:54.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 10 monclient: _reopen_session rank -1 2024-04-19T01:46:54.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-19T01:46:54.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c980a47e0 0x7f6c981cf910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:54.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 10 monclient(hunting): picked mon.noname-a con 0x7f6c980a47e0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:46:54.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c980a5100 0x7f6c98134520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:54.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 10 monclient(hunting): picked mon.noname-c con 0x7f6c980a5100 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:46:54.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c981357c0 0x7f6c98134a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:54.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 10 monclient(hunting): picked mon.noname-b con 0x7f6c981357c0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:46:54.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:54.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9809c8e0 con 0x7f6c980a47e0 2024-04-19T01:46:54.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:54.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9809c760 con 0x7f6c981357c0 2024-04-19T01:46:54.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:54.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f6c98140360 con 0x7f6c980a5100 2024-04-19T01:46:54.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 10 monclient(hunting): _renew_subs 2024-04-19T01:46:54.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9ca1f640 10 monclient(hunting): authenticate will time out at 1200.107210s 2024-04-19T01:46:54.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c95d9b640 1 -- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c980a5100 msgr2=0x7f6c98134520 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:3301/0 2024-04-19T01:46:54.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c95d9b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c980a5100 0x7f6c98134520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:46:54.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c980a47e0 0x7f6c981cf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:54.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c980a47e0 0x7f6c981cf910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44324/0 (socket says 172.21.15.62:44324) 2024-04-19T01:46:54.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9659c640 1 -- 172.21.15.62:0/2669936956 learned_addr learned my addr 172.21.15.62:0/2669936956 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:46:54.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9659c640 10 monclient(hunting): get_auth_request con 0x7f6c980a47e0 auth_method 0 2024-04-19T01:46:54.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:54.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:54.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:54.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:54.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:54.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:54.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9659c640 10 monclient(hunting): handle_auth_done global_id 14817 payload 824 2024-04-19T01:46:54.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9659c640 1 -- 172.21.15.62:0/2669936956 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c980a5100 msgr2=0x7f6c98134520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:46:54.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9659c640 1 --2- 172.21.15.62:0/2669936956 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c980a5100 0x7f6c98134520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:54.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9659c640 1 -- 172.21.15.62:0/2669936956 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c981357c0 msgr2=0x7f6c98134a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:54.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9659c640 1 --2- 172.21.15.62:0/2669936956 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c981357c0 0x7f6c98134a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:54.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9659c640 10 monclient: _finish_hunting 0 2024-04-19T01:46:54.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9659c640 1 monclient: found mon.noname-a 2024-04-19T01:46:54.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:46:54.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:46:54.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.283+0000 7f6c9659c640 1 -- 172.21.15.62:0/2669936956 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c980726d0 con 0x7f6c980a47e0 2024-04-19T01:46:54.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.287+0000 7f6c9659c640 1 -- 172.21.15.62:0/2669936956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c980a47e0 msgr2=0x7f6c981cf910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:54.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.287+0000 7f6c9659c640 1 --2- 172.21.15.62:0/2669936956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c980a47e0 0x7f6c981cf910 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6c8801cad0 tx=0x7f6c8801aa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:54.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.287+0000 7f6c86ffd640 1 -- 172.21.15.62:0/2669936956 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c880b0050 con 0x7f6c980a47e0 2024-04-19T01:46:54.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.287+0000 7f6c86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:54.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.287+0000 7f6c86ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:46:54.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.287+0000 7f6c86ffd640 10 monclient: dump: 2024-04-19T01:46:54.295 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:54.295 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:54.295 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:54.296 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:54.296 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:54.296 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:54.296 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:54.296 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:54.296 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:54.296 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:54.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.287+0000 7f6c86ffd640 10 monclient: _finish_auth 0 2024-04-19T01:46:54.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.287+0000 7f6c86ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:46:54.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.287+0000 7f6c86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:46:54.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.287+0000 7f6c86ffd640 1 -- 172.21.15.62:0/2669936956 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c880bc070 con 0x7f6c980a47e0 2024-04-19T01:46:54.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.287+0000 7f6c9ca1f640 5 monclient: authenticate success, global_id 14817 2024-04-19T01:46:54.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.287+0000 7f6c9ca1f640 10 monclient: _renew_subs 2024-04-19T01:46:54.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.287+0000 7f6c9ca1f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:46:54.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.287+0000 7f6c9ca1f640 1 -- 172.21.15.62:0/2669936956 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c9806e820 con 0x7f6c980a47e0 2024-04-19T01:46:54.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.287+0000 7f6c86ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:46:54.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.287+0000 7f6c86ffd640 1 -- 172.21.15.62:0/2669936956 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c880b5c80 con 0x7f6c980a47e0 2024-04-19T01:46:54.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.287+0000 7f6c86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:54.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.287+0000 7f6c86ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:46:54.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.287+0000 7f6c86ffd640 10 monclient: dump: 2024-04-19T01:46:54.297 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:54.297 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:54.297 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:54.297 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:54.297 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:54.297 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:54.297 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:54.297 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:54.297 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:54.298 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:54.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.287+0000 7f6c94d99640 4 set_mon_vals no callback set 2024-04-19T01:46:54.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.287+0000 7f6c9ca1f640 10 monclient: _renew_subs 2024-04-19T01:46:54.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.287+0000 7f6c9ca1f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:46:54.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.287+0000 7f6c9ca1f640 1 -- 172.21.15.62:0/2669936956 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c98120650 con 0x7f6c980a47e0 2024-04-19T01:46:54.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.287+0000 7f6c86ffd640 1 -- 172.21.15.62:0/2669936956 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f6c880b7450 con 0x7f6c980a47e0 2024-04-19T01:46:54.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.287+0000 7f6c9ca1f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:54.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.287+0000 7f6c9ca1f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:54.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.287+0000 7f6c9ca1f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:46:54.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.287+0000 7f6c9ca1f640 1 -- 172.21.15.62:0/2669936956 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c980692b0 con 0x7f6c980a47e0 2024-04-19T01:46:54.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.287+0000 7f6c86ffd640 1 --2- 172.21.15.62:0/2669936956 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f6c78036450 0x7f6c78038910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:54.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.287+0000 7f6c86ffd640 1 -- 172.21.15.62:0/2669936956 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(225..225 src has 115..225) ==== 4346+0+0 (secure 0 0 0) 0x7f6c880e3760 con 0x7f6c980a47e0 2024-04-19T01:46:54.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.287+0000 7f6c95d9b640 1 --2- 172.21.15.62:0/2669936956 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f6c78036450 0x7f6c78038910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:54.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.287+0000 7f6c95d9b640 10 monclient: get_auth_request con 0x7f6c78036450 auth_method 0 2024-04-19T01:46:54.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.291+0000 7f6c95d9b640 1 --2- 172.21.15.62:0/2669936956 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f6c78036450 0x7f6c78038910 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6c981d0980 tx=0x7f6c8c007b30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:54.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.295+0000 7f6c86ffd640 1 -- 172.21.15.62:0/2669936956 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6c880c5020 con 0x7f6c980a47e0 2024-04-19T01:46:54.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.295+0000 7f6c86ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:54.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.295+0000 7f6c86ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:46:54.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.531+0000 7f6c9ca1f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:46:54.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.531+0000 7f6c9ca1f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:46:54.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.531+0000 7f6c9ca1f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:46:54.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.531+0000 7f6c9ca1f640 1 -- 172.21.15.62:0/2669936956 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6c9806b980 con 0x7f6c980a47e0 2024-04-19T01:46:54.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.531+0000 7f6c86ffd640 1 -- 172.21.15.62:0/2669936956 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f6c9809f7c0 con 0x7f6c980a47e0 2024-04-19T01:46:54.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.531+0000 7f6c86ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:46:54.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.531+0000 7f6c86ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:46:54.532 INFO:teuthology.orchestra.run.smithi062.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","modified":"2024-04-19T01:41:38.497795Z","created":"2024-04-19T01:41:38.497795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:46:54.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.535+0000 7f6c9ca1f640 1 -- 172.21.15.62:0/2669936956 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f6c78036450 msgr2=0x7f6c78038910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:54.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.535+0000 7f6c9ca1f640 1 --2- 172.21.15.62:0/2669936956 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f6c78036450 0x7f6c78038910 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6c981d0980 tx=0x7f6c8c007b30 comp rx=0 tx=0).stop 2024-04-19T01:46:54.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.535+0000 7f6c9ca1f640 10 monclient: shutdown 2024-04-19T01:46:54.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.535+0000 7f6c9ca1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:46:54.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.535+0000 7f6c9ca1f640 1 -- 172.21.15.62:0/2669936956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c980a47e0 msgr2=0x7f6c981cf910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:54.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.535+0000 7f6c9ca1f640 1 --2- 172.21.15.62:0/2669936956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c980a47e0 0x7f6c981cf910 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6c8801cad0 tx=0x7f6c8801aa70 comp rx=0 tx=0).stop 2024-04-19T01:46:54.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.535+0000 7f6c9ca1f640 1 -- 172.21.15.62:0/2669936956 shutdown_connections 2024-04-19T01:46:54.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.535+0000 7f6c9ca1f640 1 --2- 172.21.15.62:0/2669936956 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f6c78036450 0x7f6c78038910 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:54.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.535+0000 7f6c9ca1f640 1 --2- 172.21.15.62:0/2669936956 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f6c980a5100 0x7f6c98134520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:54.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.535+0000 7f6c9ca1f640 1 --2- 172.21.15.62:0/2669936956 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c981357c0 0x7f6c98134a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:54.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.535+0000 7f6c9ca1f640 1 --2- 172.21.15.62:0/2669936956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c980a47e0 0x7f6c981cf910 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:54.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.535+0000 7f6c9ca1f640 1 -- 172.21.15.62:0/2669936956 >> 172.21.15.62:0/2669936956 conn(0x7f6c980a23d0 msgr2=0x7f6c98071870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:46:54.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.535+0000 7f6c9ca1f640 1 -- 172.21.15.62:0/2669936956 shutdown_connections 2024-04-19T01:46:54.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:54.535+0000 7f6c9ca1f640 1 -- 172.21.15.62:0/2669936956 wait complete. 2024-04-19T01:46:57.553 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:46:57.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccfb3a0640 10 monclient: get_monmap_and_config 2024-04-19T01:46:57.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccfb3a0640 10 monclient: build_initial_monmap 2024-04-19T01:46:57.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccfb3a0640 10 monclient: monmap: 2024-04-19T01:46:57.686 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:46:57.686 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:57.686 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:46:57.691483+0000 2024-04-19T01:46:57.686 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:46:57.691483+0000 2024-04-19T01:46:57.686 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:46:57.686 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:57.686 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:46:57.686 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:46:57.686 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:46:57.686 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:46:57.687 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:46:57.687 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:57.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccfb3a0640 1 Processor -- start 2024-04-19T01:46:57.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccfb3a0640 1 -- start start 2024-04-19T01:46:57.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccfb3a0640 10 monclient: init 2024-04-19T01:46:57.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccfb3a0640 10 monclient: _reopen_session rank -1 2024-04-19T01:46:57.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccfb3a0640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-04-19T01:46:57.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccfb3a0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fccf40a2ef0 addr v1:172.21.15.71:6789/0 2024-04-19T01:46:57.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccfb3a0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fccf413a570 0x7fccf413c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:57.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccfb3a0640 10 monclient(hunting): picked mon.noname-b con 0x7fccf413a570 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:46:57.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccfb3a0640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fccf413cea0 0x7fccf413f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:57.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccfb3a0640 10 monclient(hunting): picked mon.noname-a con 0x7fccf413cea0 addr v2:172.21.15.62:3300/0 2024-04-19T01:46:57.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccfb3a0640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccf4137550 con 0x7fccf40a2ef0 2024-04-19T01:46:57.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccfb3a0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:57.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccfb3a0640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fccf409bd90 con 0x7fccf413cea0 2024-04-19T01:46:57.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccfb3a0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:57.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccfb3a0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fccf409c070 con 0x7fccf413a570 2024-04-19T01:46:57.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccfb3a0640 10 monclient(hunting): _renew_subs 2024-04-19T01:46:57.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccfb3a0640 10 monclient(hunting): authenticate will time out at 1203.511168s 2024-04-19T01:46:57.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccf893b640 1 -- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fccf413a570 msgr2=0x7fccf413c960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.62:3301/0 2024-04-19T01:46:57.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccf893b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fccf413a570 0x7fccf413c960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:46:57.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccf993d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fccf413cea0 0x7fccf413f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:57.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccf993d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fccf413cea0 0x7fccf413f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59494/0 (socket says 172.21.15.62:59494) 2024-04-19T01:46:57.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccf993d640 1 -- 172.21.15.62:0/187103765 learned_addr learned my addr 172.21.15.62:0/187103765 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:46:57.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccf993d640 10 monclient(hunting): get_auth_request con 0x7fccf413cea0 auth_method 0 2024-04-19T01:46:57.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccf993d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:57.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccf993d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:57.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccf993d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:57.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccf993d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:57.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccf993d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:57.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccf993d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:57.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccf993d640 10 monclient(hunting): handle_auth_done global_id 14820 payload 274 2024-04-19T01:46:57.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccf993d640 1 -- 172.21.15.62:0/187103765 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fccf413a570 msgr2=0x7fccf413c960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:46:57.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccf993d640 1 --2- 172.21.15.62:0/187103765 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fccf413a570 0x7fccf413c960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:57.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccf993d640 1 -- 172.21.15.62:0/187103765 >> v1:172.21.15.71:6789/0 conn(0x7fccf40a2ef0 legacy=0x7fccf41379a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:46:57.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccf993d640 10 monclient: _finish_hunting 0 2024-04-19T01:46:57.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccf993d640 1 monclient: found mon.noname-a 2024-04-19T01:46:57.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccf993d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:46:57.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccf993d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:46:57.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.687+0000 7fccf993d640 1 -- 172.21.15.62:0/187103765 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccf409c350 con 0x7fccf413cea0 2024-04-19T01:46:57.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccf993d640 1 -- 172.21.15.62:0/187103765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccf413cea0 msgr2=0x7fccf413f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:57.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccf993d640 1 --2- 172.21.15.62:0/187103765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccf413cea0 0x7fccf413f290 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fccec009e50 tx=0x7fccec01c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a59beb22a8675689 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:57.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccebfff640 1 -- 172.21.15.62:0/187103765 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccec0a5680 con 0x7fccf413cea0 2024-04-19T01:46:57.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:57.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccebfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:46:57.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccebfff640 10 monclient: dump: 2024-04-19T01:46:57.693 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:57.693 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:57.693 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:57.693 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:57.693 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:57.694 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:57.694 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:57.694 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:57.694 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:57.694 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:57.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccebfff640 10 monclient: _finish_auth 0 2024-04-19T01:46:57.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccebfff640 10 monclient: _check_auth_tickets 2024-04-19T01:46:57.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:46:57.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccebfff640 1 -- 172.21.15.62:0/187103765 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccec0a5820 con 0x7fccf413cea0 2024-04-19T01:46:57.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 5 monclient: authenticate success, global_id 14820 2024-04-19T01:46:57.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:46:57.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccebfff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:46:57.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 10 monclient: get_monmap_and_config success 2024-04-19T01:46:57.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccebfff640 1 -- 172.21.15.62:0/187103765 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccec0a5bf0 con 0x7fccf413cea0 2024-04-19T01:46:57.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 4 set_mon_vals no callback set 2024-04-19T01:46:57.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:57.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:46:57.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccebfff640 10 monclient: dump: 2024-04-19T01:46:57.695 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:57.695 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:57.695 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:57.695 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:57.695 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:57.695 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:57.695 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:57.695 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:57.695 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:57.695 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:57.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 10 monclient: shutdown 2024-04-19T01:46:57.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:46:57.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 1 -- 172.21.15.62:0/187103765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccf413cea0 msgr2=0x7fccf413f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:57.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 1 --2- 172.21.15.62:0/187103765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccf413cea0 0x7fccf413f290 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fccec009e50 tx=0x7fccec01c8b0 comp rx=0 tx=0).stop 2024-04-19T01:46:57.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 1 -- 172.21.15.62:0/187103765 shutdown_connections 2024-04-19T01:46:57.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 1 --2- 172.21.15.62:0/187103765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccf413cea0 0x7fccf413f290 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:57.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 1 --2- 172.21.15.62:0/187103765 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fccf413a570 0x7fccf413c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:57.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 1 -- 172.21.15.62:0/187103765 >> 172.21.15.62:0/187103765 conn(0x7fccf40a23d0 msgr2=0x7fccf40a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:46:57.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 1 -- 172.21.15.62:0/187103765 shutdown_connections 2024-04-19T01:46:57.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 1 -- 172.21.15.62:0/187103765 wait complete. 2024-04-19T01:46:57.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 10 monclient: build_initial_monmap 2024-04-19T01:46:57.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 10 monclient: monmap: 2024-04-19T01:46:57.701 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:46:57.701 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:46:57.701 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:46:57.701 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:46:57.701 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:46:57.701 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:57.701 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:46:57.701 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:46:57.702 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:46:57.702 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:57.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 1 Processor -- start 2024-04-19T01:46:57.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 1 -- start start 2024-04-19T01:46:57.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 10 monclient: init 2024-04-19T01:46:57.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 10 monclient: _reopen_session rank -1 2024-04-19T01:46:57.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-19T01:46:57.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fccf40a2ef0 0x7fccf41cf180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:57.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 10 monclient(hunting): picked mon.noname-b con 0x7fccf40a2ef0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:46:57.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccf413a570 0x7fccf4136120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:57.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 10 monclient(hunting): picked mon.noname-a con 0x7fccf413a570 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:46:57.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fccf413cea0 0x7fccf411cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:57.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 10 monclient(hunting): picked mon.noname-c con 0x7fccf413cea0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:46:57.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:57.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fccf4141ff0 con 0x7fccf413a570 2024-04-19T01:46:57.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:57.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fccf4072520 con 0x7fccf40a2ef0 2024-04-19T01:46:57.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:46:57.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fccf4142170 con 0x7fccf413cea0 2024-04-19T01:46:57.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 10 monclient(hunting): _renew_subs 2024-04-19T01:46:57.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccfb3a0640 10 monclient(hunting): authenticate will time out at 1203.518539s 2024-04-19T01:46:57.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccf993d640 1 -- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fccf413cea0 msgr2=0x7fccf411cb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:3301/0 2024-04-19T01:46:57.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccf993d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fccf413cea0 0x7fccf411cb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:46:57.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccf893b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccf413a570 0x7fccf4136120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:57.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccf893b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccf413a570 0x7fccf4136120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59504/0 (socket says 172.21.15.62:59504) 2024-04-19T01:46:57.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccf893b640 1 -- 172.21.15.62:0/3295190629 learned_addr learned my addr 172.21.15.62:0/3295190629 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:46:57.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccf893b640 10 monclient(hunting): get_auth_request con 0x7fccf413a570 auth_method 0 2024-04-19T01:46:57.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccf893b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:57.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccf893b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:57.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccf893b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:57.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccf913c640 1 --2- 172.21.15.62:0/3295190629 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fccf40a2ef0 0x7fccf41cf180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:57.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccf913c640 10 monclient(hunting): get_auth_request con 0x7fccf40a2ef0 auth_method 0 2024-04-19T01:46:57.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccf913c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:46:57.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccf913c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:46:57.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccf913c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:46:57.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccf893b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:57.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccf893b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:57.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccf893b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:57.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccf913c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:46:57.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccf913c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:46:57.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccf913c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:46:57.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccf893b640 10 monclient(hunting): handle_auth_done global_id 14823 payload 824 2024-04-19T01:46:57.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccf893b640 1 -- 172.21.15.62:0/3295190629 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fccf413cea0 msgr2=0x7fccf411cb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:46:57.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccf893b640 1 --2- 172.21.15.62:0/3295190629 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fccf413cea0 0x7fccf411cb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:57.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccf893b640 1 -- 172.21.15.62:0/3295190629 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fccf40a2ef0 msgr2=0x7fccf41cf180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:57.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccf893b640 1 --2- 172.21.15.62:0/3295190629 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fccf40a2ef0 0x7fccf41cf180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:57.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccf893b640 10 monclient: _finish_hunting 0 2024-04-19T01:46:57.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccf893b640 1 monclient: found mon.noname-a 2024-04-19T01:46:57.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccf893b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:46:57.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccf893b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:46:57.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.695+0000 7fccf893b640 1 -- 172.21.15.62:0/3295190629 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccf411e7c0 con 0x7fccf413a570 2024-04-19T01:46:57.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.699+0000 7fccf893b640 1 -- 172.21.15.62:0/3295190629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccf413a570 msgr2=0x7fccf4136120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:46:57.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.699+0000 7fccf893b640 1 --2- 172.21.15.62:0/3295190629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccf413a570 0x7fccf4136120 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fccf000b550 tx=0x7fccf000ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:57.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.699+0000 7fcce9ffb640 1 -- 172.21.15.62:0/3295190629 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf00050f0 con 0x7fccf413a570 2024-04-19T01:46:57.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.699+0000 7fcce9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:57.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.699+0000 7fcce9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:46:57.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.699+0000 7fcce9ffb640 10 monclient: dump: 2024-04-19T01:46:57.705 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:57.706 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:57.706 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:57.706 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:57.706 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:57.706 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:57.706 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:57.706 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:57.706 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:57.706 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:57.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.699+0000 7fcce9ffb640 10 monclient: _finish_auth 0 2024-04-19T01:46:57.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.699+0000 7fcce9ffb640 10 monclient: _check_auth_tickets 2024-04-19T01:46:57.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.699+0000 7fcce9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:46:57.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.699+0000 7fcce9ffb640 1 -- 172.21.15.62:0/3295190629 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccf0015070 con 0x7fccf413a570 2024-04-19T01:46:57.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.699+0000 7fccfb3a0640 5 monclient: authenticate success, global_id 14823 2024-04-19T01:46:57.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.699+0000 7fccfb3a0640 10 monclient: _renew_subs 2024-04-19T01:46:57.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.699+0000 7fccfb3a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:46:57.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.699+0000 7fccfb3a0640 1 -- 172.21.15.62:0/3295190629 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccf41d0070 con 0x7fccf413a570 2024-04-19T01:46:57.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.699+0000 7fcce9ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:46:57.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.699+0000 7fcce9ffb640 1 -- 172.21.15.62:0/3295190629 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf00054d0 con 0x7fccf413a570 2024-04-19T01:46:57.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.699+0000 7fcce9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:46:57.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.699+0000 7fcce9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:46:57.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.699+0000 7fcce9ffb640 10 monclient: dump: 2024-04-19T01:46:57.707 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:46:57.707 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:46:57.707 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:57.707 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:46:57.707 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:46:57.707 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:46:57.707 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:46:57.707 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:46:57.707 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:46:57.707 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:46:57.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.699+0000 7fcceb7fe640 4 set_mon_vals no callback set 2024-04-19T01:46:57.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.699+0000 7fccfb3a0640 10 monclient: _renew_subs 2024-04-19T01:46:57.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.699+0000 7fccfb3a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:46:57.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.699+0000 7fccfb3a0640 1 -- 172.21.15.62:0/3295190629 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccf4120210 con 0x7fccf413a570 2024-04-19T01:46:57.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.699+0000 7fccfb3a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:57.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.699+0000 7fccfb3a0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:57.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.699+0000 7fccfb3a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:46:57.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.699+0000 7fccfb3a0640 1 -- 172.21.15.62:0/3295190629 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccf40692b0 con 0x7fccf413a570 2024-04-19T01:46:57.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.699+0000 7fcce9ffb640 1 -- 172.21.15.62:0/3295190629 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fccf0005670 con 0x7fccf413a570 2024-04-19T01:46:57.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.699+0000 7fcce9ffb640 1 --2- 172.21.15.62:0/3295190629 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fccd80363d0 0x7fccd8038890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:46:57.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.699+0000 7fcce9ffb640 1 -- 172.21.15.62:0/3295190629 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(225..225 src has 115..225) ==== 4346+0+0 (secure 0 0 0) 0x7fccf0049120 con 0x7fccf413a570 2024-04-19T01:46:57.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.699+0000 7fccf913c640 1 --2- 172.21.15.62:0/3295190629 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fccd80363d0 0x7fccd8038890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:46:57.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.699+0000 7fccf913c640 10 monclient: get_auth_request con 0x7fccd80363d0 auth_method 0 2024-04-19T01:46:57.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.703+0000 7fccf913c640 1 --2- 172.21.15.62:0/3295190629 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fccd80363d0 0x7fccd8038890 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fccf411e3c0 tx=0x7fcce4009060 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:46:57.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.703+0000 7fcce9ffb640 1 -- 172.21.15.62:0/3295190629 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fccf001a460 con 0x7fccf413a570 2024-04-19T01:46:57.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.703+0000 7fcce9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:46:57.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.703+0000 7fcce9ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:46:57.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.927+0000 7fccfb3a0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:46:57.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.927+0000 7fccfb3a0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:46:57.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.927+0000 7fccfb3a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:46:57.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.927+0000 7fccfb3a0640 1 -- 172.21.15.62:0/3295190629 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fccf406b980 con 0x7fccf413a570 2024-04-19T01:46:57.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.927+0000 7fcce9ffb640 1 -- 172.21.15.62:0/3295190629 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fccf0018360 con 0x7fccf413a570 2024-04-19T01:46:57.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.927+0000 7fcce9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:46:57.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.927+0000 7fcce9ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:46:57.927 INFO:teuthology.orchestra.run.smithi062.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","modified":"2024-04-19T01:41:38.497795Z","created":"2024-04-19T01:41:38.497795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:46:57.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.931+0000 7fccfb3a0640 1 -- 172.21.15.62:0/3295190629 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fccd80363d0 msgr2=0x7fccd8038890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:57.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.931+0000 7fccfb3a0640 1 --2- 172.21.15.62:0/3295190629 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fccd80363d0 0x7fccd8038890 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fccf411e3c0 tx=0x7fcce4009060 comp rx=0 tx=0).stop 2024-04-19T01:46:57.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.931+0000 7fccfb3a0640 10 monclient: shutdown 2024-04-19T01:46:57.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.931+0000 7fccfb3a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:46:57.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.931+0000 7fccfb3a0640 1 -- 172.21.15.62:0/3295190629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccf413a570 msgr2=0x7fccf4136120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:46:57.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.931+0000 7fccfb3a0640 1 --2- 172.21.15.62:0/3295190629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccf413a570 0x7fccf4136120 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fccf000b550 tx=0x7fccf000ba10 comp rx=0 tx=0).stop 2024-04-19T01:46:57.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.931+0000 7fccfb3a0640 1 -- 172.21.15.62:0/3295190629 shutdown_connections 2024-04-19T01:46:57.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.931+0000 7fccfb3a0640 1 --2- 172.21.15.62:0/3295190629 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fccd80363d0 0x7fccd8038890 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:57.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.931+0000 7fccfb3a0640 1 --2- 172.21.15.62:0/3295190629 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fccf413cea0 0x7fccf411cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:57.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.931+0000 7fccfb3a0640 1 --2- 172.21.15.62:0/3295190629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccf413a570 0x7fccf4136120 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:57.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.931+0000 7fccfb3a0640 1 --2- 172.21.15.62:0/3295190629 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fccf40a2ef0 0x7fccf41cf180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:46:57.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.931+0000 7fccfb3a0640 1 -- 172.21.15.62:0/3295190629 >> 172.21.15.62:0/3295190629 conn(0x7fccf40a23d0 msgr2=0x7fccf40a45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:46:57.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.931+0000 7fccfb3a0640 1 -- 172.21.15.62:0/3295190629 shutdown_connections 2024-04-19T01:46:57.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:46:57.931+0000 7fccfb3a0640 1 -- 172.21.15.62:0/3295190629 wait complete. 2024-04-19T01:47:00.946 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:47:01.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.079+0000 7fbd7b8b0640 10 monclient: get_monmap_and_config 2024-04-19T01:47:01.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.079+0000 7fbd7b8b0640 10 monclient: build_initial_monmap 2024-04-19T01:47:01.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.079+0000 7fbd7b8b0640 10 monclient: monmap: 2024-04-19T01:47:01.079 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:47:01.079 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:47:01.079 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:47:01.084417+0000 2024-04-19T01:47:01.079 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:47:01.084417+0000 2024-04-19T01:47:01.079 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:47:01.079 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:47:01.079 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:47:01.079 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:47:01.079 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:47:01.079 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:47:01.080 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:47:01.080 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:47:01.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.079+0000 7fbd7b8b0640 1 Processor -- start 2024-04-19T01:47:01.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.079+0000 7fbd7b8b0640 1 -- start start 2024-04-19T01:47:01.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.079+0000 7fbd7b8b0640 10 monclient: init 2024-04-19T01:47:01.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.079+0000 7fbd7b8b0640 10 monclient: _reopen_session rank -1 2024-04-19T01:47:01.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.079+0000 7fbd7b8b0640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-04-19T01:47:01.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.079+0000 7fbd7b8b0640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fbd741377e0 0x7fbd740a22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:47:01.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.079+0000 7fbd7b8b0640 10 monclient(hunting): picked mon.noname-c con 0x7fbd741377e0 addr v2:172.21.15.71:3300/0 2024-04-19T01:47:01.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.079+0000 7fbd7b8b0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd74137bc0 0x7fbd740a0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:47:01.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.079+0000 7fbd7b8b0640 10 monclient(hunting): picked mon.noname-b con 0x7fbd74137bc0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:47:01.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.079+0000 7fbd7b8b0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbd74137ff0 addr v1:172.21.15.62:6789/0 2024-04-19T01:47:01.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.079+0000 7fbd7b8b0640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd74073630 con 0x7fbd74137ff0 2024-04-19T01:47:01.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.079+0000 7fbd7b8b0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:47:01.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.079+0000 7fbd7b8b0640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fbd740a1bf0 con 0x7fbd741377e0 2024-04-19T01:47:01.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.079+0000 7fbd7b8b0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:47:01.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.079+0000 7fbd7b8b0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fbd740a2830 con 0x7fbd74137bc0 2024-04-19T01:47:01.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.079+0000 7fbd7b8b0640 10 monclient(hunting): _renew_subs 2024-04-19T01:47:01.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.079+0000 7fbd7b8b0640 10 monclient(hunting): authenticate will time out at 1206.903937s 2024-04-19T01:47:01.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.083+0000 7fbd79e4d640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fbd74137ff0 0x7fbd740a1010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37238/0 (socket says 172.21.15.62:37238) 2024-04-19T01:47:01.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.083+0000 7fbd79e4d640 1 -- 172.21.15.62:0/1529399359 learned_addr learned my addr 172.21.15.62:0/1529399359 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:47:01.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.083+0000 7fbd78e4b640 1 -- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd74137bc0 msgr2=0x7fbd740a0ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.62:3301/0 2024-04-19T01:47:01.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.083+0000 7fbd78e4b640 1 --2- 172.21.15.62:0/1529399359 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd74137bc0 0x7fbd740a0ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:47:01.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.083+0000 7fbd7964c640 1 --2- 172.21.15.62:0/1529399359 >> v2:172.21.15.71:3300/0 conn(0x7fbd741377e0 0x7fbd740a22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:47:01.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.083+0000 7fbd7964c640 10 monclient(hunting): get_auth_request con 0x7fbd741377e0 auth_method 0 2024-04-19T01:47:01.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.083+0000 7fbd7964c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:47:01.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.083+0000 7fbd7964c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:47:01.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.083+0000 7fbd7964c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:47:01.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.083+0000 7fbd7964c640 1 -- 172.21.15.62:0/1529399359 >> v2:172.21.15.71:3300/0 conn(0x7fbd741377e0 msgr2=0x7fbd740a22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:47:01.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.083+0000 7fbd7964c640 1 -- 172.21.15.62:0/1529399359 >> v2:172.21.15.71:3300/0 conn(0x7fbd741377e0 msgr2=0x7fbd740a22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:47:01.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.083+0000 7fbd7964c640 1 --2- 172.21.15.62:0/1529399359 >> v2:172.21.15.71:3300/0 conn(0x7fbd741377e0 0x7fbd740a22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:47:01.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.083+0000 7fbd7964c640 1 --2- 172.21.15.62:0/1529399359 >> v2:172.21.15.71:3300/0 conn(0x7fbd741377e0 0x7fbd740a22f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:47:01.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.083+0000 7fbd79e4d640 1 -- 172.21.15.62:0/1529399359 >> v1:172.21.15.62:6789/0 conn(0x7fbd74137ff0 legacy=0x7fbd740a1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:47:01.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.287+0000 7fbd78e4b640 1 -- 172.21.15.62:0/1529399359 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd74137bc0 msgr2=0x7fbd740a0ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.62:3301/0 2024-04-19T01:47:01.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.287+0000 7fbd78e4b640 1 --2- 172.21.15.62:0/1529399359 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd74137bc0 0x7fbd740a0ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:47:01.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.287+0000 7fbd7964c640 1 --2- 172.21.15.62:0/1529399359 >> v2:172.21.15.71:3300/0 conn(0x7fbd741377e0 0x7fbd740a22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:47:01.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.287+0000 7fbd7964c640 10 monclient(hunting): get_auth_request con 0x7fbd741377e0 auth_method 0 2024-04-19T01:47:01.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.287+0000 7fbd7964c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:47:01.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.287+0000 7fbd7964c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:47:01.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.287+0000 7fbd7964c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:47:01.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.287+0000 7fbd7964c640 1 -- 172.21.15.62:0/1529399359 >> v2:172.21.15.71:3300/0 conn(0x7fbd741377e0 msgr2=0x7fbd740a22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:47:01.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.287+0000 7fbd7964c640 1 -- 172.21.15.62:0/1529399359 >> v2:172.21.15.71:3300/0 conn(0x7fbd741377e0 msgr2=0x7fbd740a22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:47:01.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.287+0000 7fbd7964c640 1 --2- 172.21.15.62:0/1529399359 >> v2:172.21.15.71:3300/0 conn(0x7fbd741377e0 0x7fbd740a22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:47:01.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.287+0000 7fbd7964c640 1 --2- 172.21.15.62:0/1529399359 >> v2:172.21.15.71:3300/0 conn(0x7fbd741377e0 0x7fbd740a22f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:47:01.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.691+0000 7fbd78e4b640 1 -- 172.21.15.62:0/1529399359 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd74137bc0 msgr2=0x7fbd740a0ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.62:3301/0 2024-04-19T01:47:01.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.691+0000 7fbd78e4b640 1 --2- 172.21.15.62:0/1529399359 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd74137bc0 0x7fbd740a0ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:47:01.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.691+0000 7fbd7964c640 1 --2- 172.21.15.62:0/1529399359 >> v2:172.21.15.71:3300/0 conn(0x7fbd741377e0 0x7fbd740a22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:47:01.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.691+0000 7fbd7964c640 10 monclient(hunting): get_auth_request con 0x7fbd741377e0 auth_method 0 2024-04-19T01:47:01.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.691+0000 7fbd7964c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:47:01.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.691+0000 7fbd7964c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:47:01.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.691+0000 7fbd7964c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:47:01.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.691+0000 7fbd7964c640 1 -- 172.21.15.62:0/1529399359 >> v2:172.21.15.71:3300/0 conn(0x7fbd741377e0 msgr2=0x7fbd740a22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:47:01.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.691+0000 7fbd7964c640 1 -- 172.21.15.62:0/1529399359 >> v2:172.21.15.71:3300/0 conn(0x7fbd741377e0 msgr2=0x7fbd740a22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:47:01.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.691+0000 7fbd7964c640 1 --2- 172.21.15.62:0/1529399359 >> v2:172.21.15.71:3300/0 conn(0x7fbd741377e0 0x7fbd740a22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:47:01.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:01.691+0000 7fbd7964c640 1 --2- 172.21.15.62:0/1529399359 >> v2:172.21.15.71:3300/0 conn(0x7fbd741377e0 0x7fbd740a22f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:47:02.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:02.495+0000 7fbd78e4b640 1 -- 172.21.15.62:0/1529399359 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd74137bc0 msgr2=0x7fbd740a0ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.62:3301/0 2024-04-19T01:47:02.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:02.495+0000 7fbd78e4b640 1 --2- 172.21.15.62:0/1529399359 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd74137bc0 0x7fbd740a0ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:47:02.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:02.495+0000 7fbd7964c640 1 --2- 172.21.15.62:0/1529399359 >> v2:172.21.15.71:3300/0 conn(0x7fbd741377e0 0x7fbd740a22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:47:02.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:02.495+0000 7fbd7964c640 10 monclient(hunting): get_auth_request con 0x7fbd741377e0 auth_method 0 2024-04-19T01:47:02.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:02.495+0000 7fbd7964c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:47:02.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:02.495+0000 7fbd7964c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:47:02.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:02.495+0000 7fbd7964c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:47:02.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:02.495+0000 7fbd7964c640 1 -- 172.21.15.62:0/1529399359 >> v2:172.21.15.71:3300/0 conn(0x7fbd741377e0 msgr2=0x7fbd740a22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:47:02.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:02.495+0000 7fbd7964c640 1 -- 172.21.15.62:0/1529399359 >> v2:172.21.15.71:3300/0 conn(0x7fbd741377e0 msgr2=0x7fbd740a22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:47:02.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:02.495+0000 7fbd7964c640 1 --2- 172.21.15.62:0/1529399359 >> v2:172.21.15.71:3300/0 conn(0x7fbd741377e0 0x7fbd740a22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:47:02.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:02.495+0000 7fbd7964c640 1 --2- 172.21.15.62:0/1529399359 >> v2:172.21.15.71:3300/0 conn(0x7fbd741377e0 0x7fbd740a22f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:47:04.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.079+0000 7fbd6affd640 10 monclient(hunting): tick 2024-04-19T01:47:04.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.083+0000 7fbd6affd640 10 monclient(hunting): _check_auth_tickets 2024-04-19T01:47:04.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.083+0000 7fbd6affd640 1 monclient(hunting): continuing hunt 2024-04-19T01:47:04.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.083+0000 7fbd6affd640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T01:47:04.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.083+0000 7fbd6affd640 1 -- 172.21.15.62:0/1529399359 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd74137bc0 msgr2=0x7fbd740a0ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:47:04.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.083+0000 7fbd6affd640 1 --2- 172.21.15.62:0/1529399359 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd74137bc0 0x7fbd740a0ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:47:04.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.083+0000 7fbd6affd640 1 -- 172.21.15.62:0/1529399359 >> v2:172.21.15.71:3300/0 conn(0x7fbd741377e0 msgr2=0x7fbd740a22f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:47:04.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.083+0000 7fbd6affd640 1 --2- 172.21.15.62:0/1529399359 >> v2:172.21.15.71:3300/0 conn(0x7fbd741377e0 0x7fbd740a22f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:47:04.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.083+0000 7fbd6affd640 1 -- 172.21.15.62:0/1529399359 >> v1:172.21.15.62:6789/0 conn(0x7fbd74137ff0 legacy=0x7fbd740a1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:47:04.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.083+0000 7fbd6affd640 10 monclient: _add_conns ranks=[1,3] 2024-04-19T01:47:04.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.083+0000 7fbd6affd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbd58003750 addr v1:172.21.15.71:6789/0 2024-04-19T01:47:04.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.083+0000 7fbd6affd640 1 --2- 172.21.15.62:0/1529399359 >> v2:172.21.15.62:3300/0 conn(0x7fbd580074a0 0x7fbd58009960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:47:04.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.083+0000 7fbd6affd640 10 monclient(hunting): picked mon.noname-a con 0x7fbd580074a0 addr v2:172.21.15.62:3300/0 2024-04-19T01:47:04.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.083+0000 7fbd6affd640 1 -- 172.21.15.62:0/1529399359 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd7409cce0 con 0x7fbd58003750 2024-04-19T01:47:04.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.083+0000 7fbd6affd640 10 monclient(hunting): start opening mon connection 2024-04-19T01:47:04.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.083+0000 7fbd6affd640 1 -- 172.21.15.62:0/1529399359 --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fbd740a1bf0 con 0x7fbd580074a0 2024-04-19T01:47:04.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.083+0000 7fbd6affd640 10 monclient(hunting): _renew_subs 2024-04-19T01:47:04.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.083+0000 7fbd78e4b640 1 --2- 172.21.15.62:0/1529399359 >> v2:172.21.15.62:3300/0 conn(0x7fbd580074a0 0x7fbd58009960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:47:04.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.083+0000 7fbd78e4b640 10 monclient(hunting): get_auth_request con 0x7fbd580074a0 auth_method 0 2024-04-19T01:47:04.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.083+0000 7fbd78e4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:47:04.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.083+0000 7fbd78e4b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:47:04.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.083+0000 7fbd78e4b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:47:04.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.083+0000 7fbd78e4b640 1 -- 172.21.15.62:0/1529399359 >> v2:172.21.15.62:3300/0 conn(0x7fbd580074a0 msgr2=0x7fbd58009960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:47:04.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.083+0000 7fbd78e4b640 1 -- 172.21.15.62:0/1529399359 >> v2:172.21.15.62:3300/0 conn(0x7fbd580074a0 msgr2=0x7fbd58009960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:47:04.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.083+0000 7fbd78e4b640 1 --2- 172.21.15.62:0/1529399359 >> v2:172.21.15.62:3300/0 conn(0x7fbd580074a0 0x7fbd58009960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:47:04.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.083+0000 7fbd78e4b640 1 --2- 172.21.15.62:0/1529399359 >> v2:172.21.15.62:3300/0 conn(0x7fbd580074a0 0x7fbd58009960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:47:04.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.083+0000 7fbd7964c640 1 -- 172.21.15.62:0/1529399359 >> v1:172.21.15.71:6789/0 conn(0x7fbd58003750 legacy=0x7fbd58005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:47:04.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.287+0000 7fbd78e4b640 1 --2- 172.21.15.62:0/1529399359 >> v2:172.21.15.62:3300/0 conn(0x7fbd580074a0 0x7fbd58009960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:47:04.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.287+0000 7fbd78e4b640 10 monclient(hunting): get_auth_request con 0x7fbd580074a0 auth_method 0 2024-04-19T01:47:04.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.287+0000 7fbd78e4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:47:04.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.287+0000 7fbd78e4b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:47:04.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.287+0000 7fbd78e4b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:47:04.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.287+0000 7fbd78e4b640 1 -- 172.21.15.62:0/1529399359 >> v2:172.21.15.62:3300/0 conn(0x7fbd580074a0 msgr2=0x7fbd58009960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:47:04.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.287+0000 7fbd78e4b640 1 -- 172.21.15.62:0/1529399359 >> v2:172.21.15.62:3300/0 conn(0x7fbd580074a0 msgr2=0x7fbd58009960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:47:04.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.287+0000 7fbd78e4b640 1 --2- 172.21.15.62:0/1529399359 >> v2:172.21.15.62:3300/0 conn(0x7fbd580074a0 0x7fbd58009960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:47:04.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.287+0000 7fbd78e4b640 1 --2- 172.21.15.62:0/1529399359 >> v2:172.21.15.62:3300/0 conn(0x7fbd580074a0 0x7fbd58009960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:47:04.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.691+0000 7fbd78e4b640 1 --2- 172.21.15.62:0/1529399359 >> v2:172.21.15.62:3300/0 conn(0x7fbd580074a0 0x7fbd58009960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:47:04.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.691+0000 7fbd78e4b640 10 monclient(hunting): get_auth_request con 0x7fbd580074a0 auth_method 0 2024-04-19T01:47:04.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.691+0000 7fbd78e4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:47:04.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.691+0000 7fbd78e4b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:47:04.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.691+0000 7fbd78e4b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:47:04.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.691+0000 7fbd78e4b640 1 -- 172.21.15.62:0/1529399359 >> v2:172.21.15.62:3300/0 conn(0x7fbd580074a0 msgr2=0x7fbd58009960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:47:04.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.691+0000 7fbd78e4b640 1 -- 172.21.15.62:0/1529399359 >> v2:172.21.15.62:3300/0 conn(0x7fbd580074a0 msgr2=0x7fbd58009960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:47:04.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.691+0000 7fbd78e4b640 1 --2- 172.21.15.62:0/1529399359 >> v2:172.21.15.62:3300/0 conn(0x7fbd580074a0 0x7fbd58009960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:47:04.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:04.691+0000 7fbd78e4b640 1 --2- 172.21.15.62:0/1529399359 >> v2:172.21.15.62:3300/0 conn(0x7fbd580074a0 0x7fbd58009960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:47:05.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.495+0000 7fbd78e4b640 1 --2- 172.21.15.62:0/1529399359 >> v2:172.21.15.62:3300/0 conn(0x7fbd580074a0 0x7fbd58009960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:47:05.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.495+0000 7fbd78e4b640 10 monclient(hunting): get_auth_request con 0x7fbd580074a0 auth_method 0 2024-04-19T01:47:05.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.495+0000 7fbd78e4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:47:05.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.495+0000 7fbd78e4b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:47:05.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.495+0000 7fbd78e4b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:47:05.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.495+0000 7fbd78e4b640 1 -- 172.21.15.62:0/1529399359 >> v2:172.21.15.62:3300/0 conn(0x7fbd580074a0 msgr2=0x7fbd58009960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:47:05.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.495+0000 7fbd78e4b640 1 -- 172.21.15.62:0/1529399359 >> v2:172.21.15.62:3300/0 conn(0x7fbd580074a0 msgr2=0x7fbd58009960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:47:05.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.495+0000 7fbd78e4b640 1 --2- 172.21.15.62:0/1529399359 >> v2:172.21.15.62:3300/0 conn(0x7fbd580074a0 0x7fbd58009960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:47:05.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.495+0000 7fbd78e4b640 1 --2- 172.21.15.62:0/1529399359 >> v2:172.21.15.62:3300/0 conn(0x7fbd580074a0 0x7fbd58009960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:47:05.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.567+0000 7fbd6bfff640 1 -- 172.21.15.62:0/1529399359 <== mon.1 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1530546436 0 0) 0x7fbd7409cce0 con 0x7fbd58003750 2024-04-19T01:47:05.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.567+0000 7fbd6bfff640 10 monclient(hunting): _negotiate 2024-04-19T01:47:05.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.567+0000 7fbd6bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:47:05.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.567+0000 7fbd6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:47:05.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.567+0000 7fbd6bfff640 10 monclient(hunting): my global_id is 24139 2024-04-19T01:47:05.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.567+0000 7fbd6bfff640 1 -- 172.21.15.62:0/1529399359 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd60003530 con 0x7fbd58003750 2024-04-19T01:47:05.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.567+0000 7fbd6bfff640 1 -- 172.21.15.62:0/1529399359 <== mon.1 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 372158089 0 0) 0x7fbd60003530 con 0x7fbd58003750 2024-04-19T01:47:05.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.567+0000 7fbd6bfff640 1 -- 172.21.15.62:0/1529399359 >> v2:172.21.15.62:3300/0 conn(0x7fbd580074a0 msgr2=0x7fbd58009960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:47:05.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.567+0000 7fbd6bfff640 1 --2- 172.21.15.62:0/1529399359 >> v2:172.21.15.62:3300/0 conn(0x7fbd580074a0 0x7fbd58009960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:47:05.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.567+0000 7fbd6bfff640 10 monclient: _finish_hunting 0 2024-04-19T01:47:05.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.567+0000 7fbd6bfff640 1 monclient: found mon.noname-c-legacy 2024-04-19T01:47:05.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.567+0000 7fbd6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:47:05.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.567+0000 7fbd6bfff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.71:6789/0 2024-04-19T01:47:05.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.567+0000 7fbd6bfff640 1 -- 172.21.15.62:0/1529399359 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd5800b030 con 0x7fbd58003750 2024-04-19T01:47:05.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.567+0000 7fbd6bfff640 10 monclient: _finish_auth 0 2024-04-19T01:47:05.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.567+0000 7fbd6bfff640 10 monclient: _check_auth_tickets 2024-04-19T01:47:05.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.567+0000 7fbd6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:47:05.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.567+0000 7fbd6bfff640 1 -- 172.21.15.62:0/1529399359 <== mon.1 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1244427781 0 0) 0x7fbd6400f040 con 0x7fbd58003750 2024-04-19T01:47:05.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.567+0000 7fbd6bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:47:05.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd6bfff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-19T01:47:05.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd6bfff640 10 monclient: dump: 2024-04-19T01:47:05.572 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:47:05.572 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:47:05.572 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:05.572 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:05.572 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:47:05.572 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:47:05.572 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:47:05.572 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:47:05.573 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:47:05.573 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:47:05.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd6bfff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] but i'm connected to v1:172.21.15.71:6789/0, reconnecting 2024-04-19T01:47:05.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd6bfff640 10 monclient: _reopen_session rank -1 2024-04-19T01:47:05.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd6bfff640 1 -- 172.21.15.62:0/1529399359 >> v1:172.21.15.71:6789/0 conn(0x7fbd58003750 legacy=0x7fbd58005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:47:05.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd6bfff640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-19T01:47:05.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd79e4d640 1 -- 172.21.15.62:0/1529399359 reap_dead start 2024-04-19T01:47:05.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd6bfff640 1 --2- 172.21.15.62:0/1529399359 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd74137bc0 0x7fbd600039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:47:05.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd6bfff640 10 monclient(hunting): picked mon.c con 0x7fbd74137bc0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:47:05.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd6bfff640 1 --2- 172.21.15.62:0/1529399359 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd60003ff0 0x7fbd60004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:47:05.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd6bfff640 10 monclient(hunting): picked mon.a con 0x7fbd60003ff0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:47:05.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd6bfff640 1 --2- 172.21.15.62:0/1529399359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbd60004da0 0x7fbd60005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:47:05.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd6bfff640 10 monclient(hunting): picked mon.b con 0x7fbd60004da0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:47:05.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd6bfff640 10 monclient(hunting): start opening mon connection 2024-04-19T01:47:05.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd6bfff640 1 -- 172.21.15.62:0/1529399359 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbd740a1bf0 con 0x7fbd60003ff0 2024-04-19T01:47:05.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd6bfff640 10 monclient(hunting): start opening mon connection 2024-04-19T01:47:05.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd6bfff640 1 -- 172.21.15.62:0/1529399359 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fbd6000bc80 con 0x7fbd60004da0 2024-04-19T01:47:05.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd6bfff640 10 monclient(hunting): start opening mon connection 2024-04-19T01:47:05.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd6bfff640 1 -- 172.21.15.62:0/1529399359 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fbd6000be80 con 0x7fbd74137bc0 2024-04-19T01:47:05.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd6bfff640 10 monclient(hunting): _renew_subs 2024-04-19T01:47:05.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd6bfff640 10 monclient(hunting): _finish_auth 0 2024-04-19T01:47:05.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd7964c640 1 -- 172.21.15.62:0/1529399359 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd74137bc0 msgr2=0x7fbd600039c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.62:3301/0 2024-04-19T01:47:05.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd7964c640 1 --2- 172.21.15.62:0/1529399359 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd74137bc0 0x7fbd600039c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:47:05.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd79e4d640 1 --2- 172.21.15.62:0/1529399359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbd60004da0 0x7fbd60005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:47:05.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd79e4d640 10 monclient(hunting): get_auth_request con 0x7fbd60004da0 auth_method 0 2024-04-19T01:47:05.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd79e4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:47:05.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd79e4d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:47:05.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd79e4d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:47:05.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd79e4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:47:05.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd79e4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:47:05.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd79e4d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T01:47:05.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd79e4d640 10 monclient(hunting): handle_auth_done global_id 24139 payload 293 2024-04-19T01:47:05.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd79e4d640 1 -- 172.21.15.62:0/1529399359 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd74137bc0 msgr2=0x7fbd600039c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:47:05.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd79e4d640 1 --2- 172.21.15.62:0/1529399359 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd74137bc0 0x7fbd600039c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:47:05.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd79e4d640 1 -- 172.21.15.62:0/1529399359 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd60003ff0 msgr2=0x7fbd60004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:47:05.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd79e4d640 1 --2- 172.21.15.62:0/1529399359 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd60003ff0 0x7fbd60004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:47:05.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd79e4d640 10 monclient: _finish_hunting 0 2024-04-19T01:47:05.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd79e4d640 1 monclient: found mon.b 2024-04-19T01:47:05.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd79e4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:47:05.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd79e4d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.71:3300/0 2024-04-19T01:47:05.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd79e4d640 1 -- 172.21.15.62:0/1529399359 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd60003530 con 0x7fbd60004da0 2024-04-19T01:47:05.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd79e4d640 10 monclient: _finish_auth 0 2024-04-19T01:47:05.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd79e4d640 10 monclient: _check_auth_tickets 2024-04-19T01:47:05.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd79e4d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:47:05.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd7b8b0640 5 monclient: authenticate success, global_id 24139 2024-04-19T01:47:05.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.571+0000 7fbd7b8b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:47:05.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.575+0000 7fbd79e4d640 1 -- 172.21.15.62:0/1529399359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbd60004da0 msgr2=0x7fbd60005220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:47:05.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.575+0000 7fbd79e4d640 1 --2- 172.21.15.62:0/1529399359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbd60004da0 0x7fbd60005220 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbd741383d0 tx=0x7fbd6c01ca20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9d3b20e0b527471 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:47:05.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.575+0000 7fbd6bfff640 1 -- 172.21.15.62:0/1529399359 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c0020b0 con 0x7fbd60004da0 2024-04-19T01:47:05.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.575+0000 7fbd6bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:47:05.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.575+0000 7fbd6bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:47:05.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.575+0000 7fbd6bfff640 10 monclient: dump: 2024-04-19T01:47:05.582 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:47:05.582 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:47:05.582 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:05.582 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:05.582 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:47:05.582 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:47:05.582 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:47:05.582 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:47:05.582 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:47:05.582 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:47:05.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.575+0000 7fbd7b8b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:47:05.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.575+0000 7fbd6bfff640 1 -- 172.21.15.62:0/1529399359 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd6c0ac070 con 0x7fbd60004da0 2024-04-19T01:47:05.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.575+0000 7fbd6bfff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:47:05.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.575+0000 7fbd7b8b0640 10 monclient: get_monmap_and_config success 2024-04-19T01:47:05.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.575+0000 7fbd7b8b0640 4 set_mon_vals no callback set 2024-04-19T01:47:05.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.575+0000 7fbd6bfff640 1 -- 172.21.15.62:0/1529399359 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c0a6920 con 0x7fbd60004da0 2024-04-19T01:47:05.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.575+0000 7fbd6bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:47:05.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.575+0000 7fbd6bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:47:05.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.575+0000 7fbd6bfff640 10 monclient: dump: 2024-04-19T01:47:05.583 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:47:05.583 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:47:05.583 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:05.583 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:05.584 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:47:05.584 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:47:05.584 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:47:05.584 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:47:05.584 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:47:05.584 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:47:05.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7b8b0640 10 monclient: shutdown 2024-04-19T01:47:05.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7b8b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:47:05.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7b8b0640 1 -- 172.21.15.62:0/1529399359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbd60004da0 msgr2=0x7fbd60005220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:47:05.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7b8b0640 1 --2- 172.21.15.62:0/1529399359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbd60004da0 0x7fbd60005220 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbd741383d0 tx=0x7fbd6c01ca20 comp rx=0 tx=0).stop 2024-04-19T01:47:05.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7b8b0640 1 -- 172.21.15.62:0/1529399359 shutdown_connections 2024-04-19T01:47:05.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7b8b0640 1 --2- 172.21.15.62:0/1529399359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbd60004da0 0x7fbd60005220 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:47:05.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7b8b0640 1 --2- 172.21.15.62:0/1529399359 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd60003ff0 0x7fbd60004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:47:05.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7b8b0640 1 --2- 172.21.15.62:0/1529399359 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd74137bc0 0x7fbd600039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:47:05.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7b8b0640 1 -- 172.21.15.62:0/1529399359 >> 172.21.15.62:0/1529399359 conn(0x7fbd740a2bd0 msgr2=0x7fbd740a2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:47:05.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7b8b0640 1 -- 172.21.15.62:0/1529399359 shutdown_connections 2024-04-19T01:47:05.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7b8b0640 1 -- 172.21.15.62:0/1529399359 wait complete. 2024-04-19T01:47:05.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7b8b0640 10 monclient: build_initial_monmap 2024-04-19T01:47:05.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7b8b0640 10 monclient: monmap: 2024-04-19T01:47:05.585 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:47:05.585 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:47:05.585 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:47:05.585 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:47:05.585 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:47:05.585 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:47:05.585 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:47:05.585 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:47:05.586 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:47:05.586 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:47:05.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7b8b0640 1 Processor -- start 2024-04-19T01:47:05.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7b8b0640 1 -- start start 2024-04-19T01:47:05.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7b8b0640 10 monclient: init 2024-04-19T01:47:05.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7b8b0640 10 monclient: _reopen_session rank -1 2024-04-19T01:47:05.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7b8b0640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-19T01:47:05.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7b8b0640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbd60003ff0 0x7fbd74132830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:47:05.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7b8b0640 10 monclient(hunting): picked mon.noname-b con 0x7fbd60003ff0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:47:05.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7b8b0640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd60004da0 0x7fbd74132d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:47:05.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7b8b0640 10 monclient(hunting): picked mon.noname-c con 0x7fbd60004da0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:47:05.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7b8b0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd74137bc0 0x7fbd741332b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:47:05.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7b8b0640 10 monclient(hunting): picked mon.noname-a con 0x7fbd74137bc0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:47:05.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7b8b0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:47:05.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7b8b0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbd740a1670 con 0x7fbd74137bc0 2024-04-19T01:47:05.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7b8b0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:47:05.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7b8b0640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fbd740a14f0 con 0x7fbd60003ff0 2024-04-19T01:47:05.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7b8b0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:47:05.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7b8b0640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fbd740a17f0 con 0x7fbd60004da0 2024-04-19T01:47:05.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7b8b0640 10 monclient(hunting): _renew_subs 2024-04-19T01:47:05.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7b8b0640 10 monclient(hunting): authenticate will time out at 1211.403083s 2024-04-19T01:47:05.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd78e4b640 1 -- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd60004da0 msgr2=0x7fbd74132d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:3301/0 2024-04-19T01:47:05.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd78e4b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd60004da0 0x7fbd74132d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:47:05.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd79e4d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd74137bc0 0x7fbd741332b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:47:05.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd79e4d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd74137bc0 0x7fbd741332b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35560/0 (socket says 172.21.15.62:35560) 2024-04-19T01:47:05.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd79e4d640 1 -- 172.21.15.62:0/3959096096 learned_addr learned my addr 172.21.15.62:0/3959096096 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:47:05.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd79e4d640 10 monclient(hunting): get_auth_request con 0x7fbd74137bc0 auth_method 0 2024-04-19T01:47:05.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd79e4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:47:05.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd79e4d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:47:05.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd79e4d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:47:05.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7964c640 1 --2- 172.21.15.62:0/3959096096 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbd60003ff0 0x7fbd74132830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:47:05.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7964c640 10 monclient(hunting): get_auth_request con 0x7fbd60003ff0 auth_method 0 2024-04-19T01:47:05.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7964c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:47:05.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7964c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:47:05.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7964c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:47:05.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd79e4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:47:05.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd79e4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:47:05.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd79e4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:47:05.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7964c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:47:05.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7964c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:47:05.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd7964c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:47:05.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd79e4d640 10 monclient(hunting): handle_auth_done global_id 14826 payload 824 2024-04-19T01:47:05.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd79e4d640 1 -- 172.21.15.62:0/3959096096 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd60004da0 msgr2=0x7fbd74132d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:47:05.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd79e4d640 1 --2- 172.21.15.62:0/3959096096 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd60004da0 0x7fbd74132d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:47:05.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd79e4d640 1 -- 172.21.15.62:0/3959096096 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbd60003ff0 msgr2=0x7fbd74132830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:47:05.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd79e4d640 1 --2- 172.21.15.62:0/3959096096 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbd60003ff0 0x7fbd74132830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:47:05.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd79e4d640 10 monclient: _finish_hunting 0 2024-04-19T01:47:05.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd79e4d640 1 monclient: found mon.noname-a 2024-04-19T01:47:05.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd79e4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:47:05.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd79e4d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:47:05.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.579+0000 7fbd79e4d640 1 -- 172.21.15.62:0/3959096096 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd7411d690 con 0x7fbd74137bc0 2024-04-19T01:47:05.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.583+0000 7fbd79e4d640 1 -- 172.21.15.62:0/3959096096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd74137bc0 msgr2=0x7fbd741332b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:47:05.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.583+0000 7fbd79e4d640 1 --2- 172.21.15.62:0/3959096096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd74137bc0 0x7fbd741332b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c01cf00 tx=0x7fbd60005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:47:05.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.583+0000 7fbd69ffb640 1 -- 172.21.15.62:0/3959096096 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c0a6910 con 0x7fbd74137bc0 2024-04-19T01:47:05.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.583+0000 7fbd69ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:47:05.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.583+0000 7fbd69ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:47:05.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.583+0000 7fbd69ffb640 10 monclient: dump: 2024-04-19T01:47:05.592 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:47:05.592 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:47:05.592 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:05.592 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:05.592 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:47:05.593 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:47:05.593 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:47:05.593 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:47:05.593 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:47:05.593 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:47:05.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.583+0000 7fbd69ffb640 10 monclient: _finish_auth 0 2024-04-19T01:47:05.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.583+0000 7fbd69ffb640 10 monclient: _check_auth_tickets 2024-04-19T01:47:05.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.583+0000 7fbd69ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:47:05.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.583+0000 7fbd69ffb640 1 -- 172.21.15.62:0/3959096096 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd6c0b4030 con 0x7fbd74137bc0 2024-04-19T01:47:05.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.583+0000 7fbd69ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:47:05.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.583+0000 7fbd69ffb640 1 -- 172.21.15.62:0/3959096096 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c0a6cd0 con 0x7fbd74137bc0 2024-04-19T01:47:05.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.583+0000 7fbd7b8b0640 5 monclient: authenticate success, global_id 14826 2024-04-19T01:47:05.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.583+0000 7fbd69ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:47:05.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.583+0000 7fbd69ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:47:05.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.583+0000 7fbd69ffb640 10 monclient: dump: 2024-04-19T01:47:05.594 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:47:05.594 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:47:05.594 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:05.594 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:05.594 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:47:05.594 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:47:05.594 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:47:05.594 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:47:05.594 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:47:05.594 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:47:05.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.583+0000 7fbd7b8b0640 10 monclient: _renew_subs 2024-04-19T01:47:05.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.583+0000 7fbd6b7fe640 4 set_mon_vals no callback set 2024-04-19T01:47:05.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.583+0000 7fbd7b8b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:47:05.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.583+0000 7fbd7b8b0640 1 -- 172.21.15.62:0/3959096096 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd74134c90 con 0x7fbd74137bc0 2024-04-19T01:47:05.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.583+0000 7fbd7b8b0640 10 monclient: _renew_subs 2024-04-19T01:47:05.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.583+0000 7fbd7b8b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:47:05.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.583+0000 7fbd7b8b0640 1 -- 172.21.15.62:0/3959096096 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd7411d970 con 0x7fbd74137bc0 2024-04-19T01:47:05.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.587+0000 7fbd69ffb640 1 -- 172.21.15.62:0/3959096096 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fbd6c0b03f0 con 0x7fbd74137bc0 2024-04-19T01:47:05.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.587+0000 7fbd7b8b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:47:05.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.587+0000 7fbd7b8b0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:47:05.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.587+0000 7fbd7b8b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:47:05.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.587+0000 7fbd7b8b0640 1 -- 172.21.15.62:0/3959096096 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd740692b0 con 0x7fbd74137bc0 2024-04-19T01:47:05.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.587+0000 7fbd69ffb640 1 --2- 172.21.15.62:0/3959096096 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fbd58037120 0x7fbd580395e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:47:05.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.587+0000 7fbd69ffb640 1 -- 172.21.15.62:0/3959096096 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(225..225 src has 115..225) ==== 4346+0+0 (secure 0 0 0) 0x7fbd6c0e2aa0 con 0x7fbd74137bc0 2024-04-19T01:47:05.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.587+0000 7fbd7964c640 1 --2- 172.21.15.62:0/3959096096 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fbd58037120 0x7fbd580395e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:47:05.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.587+0000 7fbd7964c640 10 monclient: get_auth_request con 0x7fbd58037120 auth_method 0 2024-04-19T01:47:05.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.587+0000 7fbd7964c640 1 --2- 172.21.15.62:0/3959096096 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fbd58037120 0x7fbd580395e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fbd6400e1f0 tx=0x7fbd6400e620 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:47:05.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.591+0000 7fbd69ffb640 1 -- 172.21.15.62:0/3959096096 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fbd6c0167d0 con 0x7fbd74137bc0 2024-04-19T01:47:05.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.591+0000 7fbd69ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:47:05.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.591+0000 7fbd69ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:47:05.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.815+0000 7fbd7b8b0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:47:05.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.815+0000 7fbd7b8b0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:47:05.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.815+0000 7fbd7b8b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:47:05.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.815+0000 7fbd7b8b0640 1 -- 172.21.15.62:0/3959096096 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbd741349f0 con 0x7fbd74137bc0 2024-04-19T01:47:05.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.819+0000 7fbd69ffb640 1 -- 172.21.15.62:0/3959096096 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7fbd6c0e1650 con 0x7fbd74137bc0 2024-04-19T01:47:05.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.819+0000 7fbd69ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:47:05.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.819+0000 7fbd69ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:47:05.817 INFO:teuthology.orchestra.run.smithi062.stdout:{"election_epoch":38,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","modified":"2024-04-19T01:41:38.497795Z","created":"2024-04-19T01:41:38.497795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:47:05.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.819+0000 7fbd7b8b0640 1 -- 172.21.15.62:0/3959096096 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fbd58037120 msgr2=0x7fbd580395e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:47:05.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.819+0000 7fbd7b8b0640 1 --2- 172.21.15.62:0/3959096096 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fbd58037120 0x7fbd580395e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fbd6400e1f0 tx=0x7fbd6400e620 comp rx=0 tx=0).stop 2024-04-19T01:47:05.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.819+0000 7fbd7b8b0640 10 monclient: shutdown 2024-04-19T01:47:05.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.819+0000 7fbd7b8b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:47:05.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.819+0000 7fbd7b8b0640 1 -- 172.21.15.62:0/3959096096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd74137bc0 msgr2=0x7fbd741332b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:47:05.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.819+0000 7fbd7b8b0640 1 --2- 172.21.15.62:0/3959096096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd74137bc0 0x7fbd741332b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c01cf00 tx=0x7fbd60005840 comp rx=0 tx=0).stop 2024-04-19T01:47:05.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.819+0000 7fbd7b8b0640 1 -- 172.21.15.62:0/3959096096 shutdown_connections 2024-04-19T01:47:05.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.819+0000 7fbd7b8b0640 1 --2- 172.21.15.62:0/3959096096 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fbd58037120 0x7fbd580395e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:47:05.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.819+0000 7fbd7b8b0640 1 --2- 172.21.15.62:0/3959096096 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbd60004da0 0x7fbd74132d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:47:05.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.819+0000 7fbd7b8b0640 1 --2- 172.21.15.62:0/3959096096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd74137bc0 0x7fbd741332b0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:47:05.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.819+0000 7fbd7b8b0640 1 --2- 172.21.15.62:0/3959096096 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbd60003ff0 0x7fbd74132830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:47:05.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.819+0000 7fbd7b8b0640 1 -- 172.21.15.62:0/3959096096 >> 172.21.15.62:0/3959096096 conn(0x7fbd740a2bd0 msgr2=0x7fbd741c1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:47:05.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.819+0000 7fbd7b8b0640 1 -- 172.21.15.62:0/3959096096 shutdown_connections 2024-04-19T01:47:05.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.819+0000 7fbd7b8b0640 1 -- 172.21.15.62:0/3959096096 wait complete. 2024-04-19T01:47:05.836 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-19T01:47:05.836 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-04-19T01:47:05.837 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-19T01:47:05.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.963+0000 7fe3dca17640 10 monclient: get_monmap_and_config 2024-04-19T01:47:05.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.963+0000 7fe3dca17640 10 monclient: build_initial_monmap 2024-04-19T01:47:05.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.963+0000 7fe3dca17640 10 monclient: monmap: 2024-04-19T01:47:05.962 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:47:05.962 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:47:05.962 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:47:05.967156+0000 2024-04-19T01:47:05.962 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:47:05.967156+0000 2024-04-19T01:47:05.962 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:47:05.962 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:47:05.963 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:47:05.963 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:47:05.963 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:47:05.963 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:47:05.963 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:47:05.963 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:47:05.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.963+0000 7fe3dca17640 1 Processor -- start 2024-04-19T01:47:05.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.963+0000 7fe3dca17640 1 -- start start 2024-04-19T01:47:05.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.963+0000 7fe3dca17640 10 monclient: init 2024-04-19T01:47:05.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.963+0000 7fe3dca17640 10 monclient: _reopen_session rank -1 2024-04-19T01:47:05.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.963+0000 7fe3dca17640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-04-19T01:47:05.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.963+0000 7fe3dca17640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe3d80a47e0 0x7fe3d809f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:47:05.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.963+0000 7fe3dca17640 10 monclient(hunting): picked mon.noname-c con 0x7fe3d80a47e0 addr v2:172.21.15.71:3300/0 2024-04-19T01:47:05.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.963+0000 7fe3dca17640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe3d809f890 addr v1:172.21.15.71:6789/0 2024-04-19T01:47:05.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.963+0000 7fe3dca17640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe3d80a52d0 addr v1:172.21.15.62:6789/0 2024-04-19T01:47:05.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.963+0000 7fe3dca17640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3d809ec50 con 0x7fe3d80a52d0 2024-04-19T01:47:05.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.963+0000 7fe3dca17640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3d809e970 con 0x7fe3d809f890 2024-04-19T01:47:05.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.963+0000 7fe3dca17640 10 monclient(hunting): start opening mon connection 2024-04-19T01:47:05.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.963+0000 7fe3dca17640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fe3d809e690 con 0x7fe3d80a47e0 2024-04-19T01:47:05.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.963+0000 7fe3dca17640 10 monclient(hunting): _renew_subs 2024-04-19T01:47:05.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.963+0000 7fe3dca17640 10 monclient(hunting): authenticate will time out at 1211.787199s 2024-04-19T01:47:05.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.963+0000 7fe3d6d9d640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fe3d80a52d0 0x7fe3d813b570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:59440/0 (socket says 172.21.15.62:59440) 2024-04-19T01:47:05.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.963+0000 7fe3d6d9d640 1 -- 172.21.15.62:0/3311351616 learned_addr learned my addr 172.21.15.62:0/3311351616 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:47:05.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.963+0000 7fe3d659c640 1 --2- 172.21.15.62:0/3311351616 >> v2:172.21.15.71:3300/0 conn(0x7fe3d80a47e0 0x7fe3d809f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:47:05.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.963+0000 7fe3d659c640 10 monclient(hunting): get_auth_request con 0x7fe3d80a47e0 auth_method 0 2024-04-19T01:47:05.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.963+0000 7fe3d659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:47:05.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.963+0000 7fe3d659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:47:05.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.963+0000 7fe3d659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:47:05.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.963+0000 7fe3d659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:47:05.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.963+0000 7fe3d659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:47:05.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.963+0000 7fe3d659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:47:05.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.967+0000 7fe3d659c640 10 monclient(hunting): handle_auth_done global_id 24145 payload 274 2024-04-19T01:47:05.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.967+0000 7fe3d659c640 1 -- 172.21.15.62:0/3311351616 >> v1:172.21.15.71:6789/0 conn(0x7fe3d809f890 legacy=0x7fe3d80a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:47:05.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.967+0000 7fe3d659c640 1 -- 172.21.15.62:0/3311351616 >> v1:172.21.15.62:6789/0 conn(0x7fe3d80a52d0 legacy=0x7fe3d813b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:47:05.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.967+0000 7fe3d659c640 10 monclient: _finish_hunting 0 2024-04-19T01:47:05.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.967+0000 7fe3d659c640 1 monclient: found mon.noname-c 2024-04-19T01:47:05.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.967+0000 7fe3d659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:47:05.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.967+0000 7fe3d659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-04-19T01:47:05.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.967+0000 7fe3d659c640 1 -- 172.21.15.62:0/3311351616 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3d80a3150 con 0x7fe3d80a47e0 2024-04-19T01:47:05.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d659c640 1 -- 172.21.15.62:0/3311351616 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3d80a47e0 msgr2=0x7fe3d809f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:47:05.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d659c640 1 --2- 172.21.15.62:0/3311351616 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3d80a47e0 0x7fe3d809f350 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe3c80097c0 tx=0x7fe3c801c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62f9ffd40a2f1d0d server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:47:05.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d5d9b640 1 -- 172.21.15.62:0/3311351616 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c80b0070 con 0x7fe3d80a47e0 2024-04-19T01:47:05.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:47:05.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:47:05.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d5d9b640 10 monclient: dump: 2024-04-19T01:47:05.970 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:47:05.970 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:47:05.970 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:05.970 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:05.970 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:47:05.970 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:47:05.970 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:47:05.970 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:47:05.970 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:47:05.970 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:47:05.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d5d9b640 10 monclient: _finish_auth 0 2024-04-19T01:47:05.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d5d9b640 10 monclient: _check_auth_tickets 2024-04-19T01:47:05.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:47:05.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d5d9b640 1 -- 172.21.15.62:0/3311351616 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3c80ab600 con 0x7fe3d80a47e0 2024-04-19T01:47:05.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d5d9b640 10 monclient: handle_config config(0 keys) 2024-04-19T01:47:05.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d5d9b640 1 -- 172.21.15.62:0/3311351616 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c80ab9d0 con 0x7fe3d80a47e0 2024-04-19T01:47:05.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 5 monclient: authenticate success, global_id 24145 2024-04-19T01:47:05.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 10 monclient: get_monmap_and_config success 2024-04-19T01:47:05.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 4 set_mon_vals no callback set 2024-04-19T01:47:05.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:47:05.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:47:05.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d5d9b640 10 monclient: dump: 2024-04-19T01:47:05.971 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:47:05.971 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:47:05.971 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:05.971 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:05.972 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:47:05.972 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:47:05.972 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:47:05.972 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:47:05.972 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:47:05.972 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:47:05.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 10 monclient: shutdown 2024-04-19T01:47:05.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:47:05.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 1 -- 172.21.15.62:0/3311351616 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3d80a47e0 msgr2=0x7fe3d809f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:47:05.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 1 --2- 172.21.15.62:0/3311351616 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3d80a47e0 0x7fe3d809f350 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe3c80097c0 tx=0x7fe3c801c3f0 comp rx=0 tx=0).stop 2024-04-19T01:47:05.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 1 -- 172.21.15.62:0/3311351616 shutdown_connections 2024-04-19T01:47:05.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 1 --2- 172.21.15.62:0/3311351616 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3d80a47e0 0x7fe3d809f350 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:47:05.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 1 -- 172.21.15.62:0/3311351616 >> 172.21.15.62:0/3311351616 conn(0x7fe3d80a23d0 msgr2=0x7fe3d80a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:47:05.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 1 -- 172.21.15.62:0/3311351616 shutdown_connections 2024-04-19T01:47:05.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 1 -- 172.21.15.62:0/3311351616 wait complete. 2024-04-19T01:47:05.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 10 monclient: build_initial_monmap 2024-04-19T01:47:05.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 10 monclient: monmap: 2024-04-19T01:47:05.977 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:47:05.977 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:47:05.977 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:47:05.977 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:47:05.977 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:47:05.977 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:47:05.977 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:47:05.977 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:47:05.977 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:47:05.977 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:47:05.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 1 Processor -- start 2024-04-19T01:47:05.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 1 -- start start 2024-04-19T01:47:05.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 10 monclient: init 2024-04-19T01:47:05.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 10 monclient: _reopen_session rank -1 2024-04-19T01:47:05.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-19T01:47:05.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3d809f890 0x7fe3d81cf630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:47:05.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 10 monclient(hunting): picked mon.noname-a con 0x7fe3d809f890 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:47:05.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3d80a47e0 0x7fe3d8139700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:47:05.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 10 monclient(hunting): picked mon.noname-b con 0x7fe3d80a47e0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:47:05.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe3d80a52d0 0x7fe3d8139c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:47:05.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 10 monclient(hunting): picked mon.noname-c con 0x7fe3d80a52d0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:47:05.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 10 monclient(hunting): start opening mon connection 2024-04-19T01:47:05.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe3d813fec0 con 0x7fe3d809f890 2024-04-19T01:47:05.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 10 monclient(hunting): start opening mon connection 2024-04-19T01:47:05.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe3d813fd40 con 0x7fe3d80a47e0 2024-04-19T01:47:05.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 10 monclient(hunting): start opening mon connection 2024-04-19T01:47:05.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fe3d8140040 con 0x7fe3d80a52d0 2024-04-19T01:47:05.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 10 monclient(hunting): _renew_subs 2024-04-19T01:47:05.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3dca17640 10 monclient(hunting): authenticate will time out at 1211.795293s 2024-04-19T01:47:05.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d6d9d640 1 -- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe3d80a52d0 msgr2=0x7fe3d8139c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:3301/0 2024-04-19T01:47:05.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d6d9d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe3d80a52d0 0x7fe3d8139c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:47:05.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3d809f890 0x7fe3d81cf630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:47:05.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3d809f890 0x7fe3d81cf630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35570/0 (socket says 172.21.15.62:35570) 2024-04-19T01:47:05.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d659c640 1 -- 172.21.15.62:0/804542104 learned_addr learned my addr 172.21.15.62:0/804542104 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:47:05.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d659c640 10 monclient(hunting): get_auth_request con 0x7fe3d809f890 auth_method 0 2024-04-19T01:47:05.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:47:05.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:47:05.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:47:05.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:47:05.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:47:05.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:47:05.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3cffff640 1 --2- 172.21.15.62:0/804542104 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3d80a47e0 0x7fe3d8139700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:47:05.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3cffff640 10 monclient(hunting): get_auth_request con 0x7fe3d80a47e0 auth_method 0 2024-04-19T01:47:05.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:47:05.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3cffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:47:05.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3cffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:47:05.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d659c640 10 monclient(hunting): handle_auth_done global_id 14829 payload 824 2024-04-19T01:47:05.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d659c640 1 -- 172.21.15.62:0/804542104 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe3d80a52d0 msgr2=0x7fe3d8139c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:47:05.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d659c640 1 --2- 172.21.15.62:0/804542104 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fe3d80a52d0 0x7fe3d8139c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:47:05.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d659c640 1 -- 172.21.15.62:0/804542104 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3d80a47e0 msgr2=0x7fe3d8139700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:47:05.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d659c640 1 --2- 172.21.15.62:0/804542104 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3d80a47e0 0x7fe3d8139700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:47:05.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d659c640 10 monclient: _finish_hunting 0 2024-04-19T01:47:05.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d659c640 1 monclient: found mon.noname-a 2024-04-19T01:47:05.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:47:05.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:47:05.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.971+0000 7fe3d659c640 1 -- 172.21.15.62:0/804542104 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3d8123dc0 con 0x7fe3d809f890 2024-04-19T01:47:05.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3d659c640 1 -- 172.21.15.62:0/804542104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3d809f890 msgr2=0x7fe3d81cf630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:47:05.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3d659c640 1 --2- 172.21.15.62:0/804542104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3d809f890 0x7fe3d81cf630 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe3c801aaa0 tx=0x7fe3c8019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:47:05.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3ceffd640 1 -- 172.21.15.62:0/804542104 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c80b9070 con 0x7fe3d809f890 2024-04-19T01:47:05.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:47:05.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3ceffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:47:05.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3ceffd640 10 monclient: dump: 2024-04-19T01:47:05.982 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:47:05.982 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:47:05.982 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:05.982 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:05.982 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:47:05.982 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:47:05.982 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:47:05.983 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:47:05.983 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:47:05.983 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:47:05.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3ceffd640 10 monclient: _finish_auth 0 2024-04-19T01:47:05.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3ceffd640 10 monclient: _check_auth_tickets 2024-04-19T01:47:05.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:47:05.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3ceffd640 1 -- 172.21.15.62:0/804542104 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3c80b5aa0 con 0x7fe3d809f890 2024-04-19T01:47:05.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3ceffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:47:05.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3ceffd640 1 -- 172.21.15.62:0/804542104 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c80b0040 con 0x7fe3d809f890 2024-04-19T01:47:05.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:47:05.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3ceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:47:05.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3ceffd640 10 monclient: dump: 2024-04-19T01:47:05.983 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:47:05.984 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:47:05.984 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:05.984 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:05.984 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:47:05.984 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:47:05.984 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:47:05.984 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:47:05.984 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:47:05.984 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:47:05.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3dca17640 5 monclient: authenticate success, global_id 14829 2024-04-19T01:47:05.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3dca17640 10 monclient: _renew_subs 2024-04-19T01:47:05.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3dca17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:47:05.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3d4d99640 4 set_mon_vals no callback set 2024-04-19T01:47:05.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3dca17640 1 -- 172.21.15.62:0/804542104 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3d8125dd0 con 0x7fe3d809f890 2024-04-19T01:47:05.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3dca17640 10 monclient: _renew_subs 2024-04-19T01:47:05.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3dca17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:47:05.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3dca17640 1 -- 172.21.15.62:0/804542104 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3d8125810 con 0x7fe3d809f890 2024-04-19T01:47:05.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3dca17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:47:05.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3dca17640 1 --2- 172.21.15.62:0/804542104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3d8122120 0x7fe3d8122500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:47:05.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3dca17640 10 monclient(hunting): start opening mon connection 2024-04-19T01:47:05.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3dca17640 1 -- 172.21.15.62:0/804542104 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe3d80692b0 con 0x7fe3d8122120 2024-04-19T01:47:05.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3ceffd640 1 -- 172.21.15.62:0/804542104 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fe3c80bc070 con 0x7fe3d809f890 2024-04-19T01:47:05.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3cffff640 1 --2- 172.21.15.62:0/804542104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3d8122120 0x7fe3d8122500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:47:05.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3cffff640 10 monclient: get_auth_request con 0x7fe3d8122120 auth_method 0 2024-04-19T01:47:05.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:47:05.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3cffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:47:05.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3cffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:47:05.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:47:05.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:47:05.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:47:05.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3ceffd640 1 --2- 172.21.15.62:0/804542104 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fe3b8036450 0x7fe3b8038910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:47:05.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3ceffd640 1 -- 172.21.15.62:0/804542104 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(225..225 src has 115..225) ==== 4346+0+0 (secure 0 0 0) 0x7fe3c80e34f0 con 0x7fe3d809f890 2024-04-19T01:47:05.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3cffff640 10 monclient(hunting): handle_auth_done global_id 14832 payload 824 2024-04-19T01:47:05.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3cffff640 1 -- 172.21.15.62:0/804542104 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe3d81259e0 con 0x7fe3d8122120 2024-04-19T01:47:05.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3d6d9d640 1 --2- 172.21.15.62:0/804542104 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fe3b8036450 0x7fe3b8038910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:47:05.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.975+0000 7fe3d6d9d640 10 monclient: get_auth_request con 0x7fe3b8036450 auth_method 0 2024-04-19T01:47:05.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.979+0000 7fe3cffff640 1 -- 172.21.15.62:0/804542104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3d8122120 msgr2=0x7fe3d8122500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:47:05.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.979+0000 7fe3cffff640 1 --2- 172.21.15.62:0/804542104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3d8122120 0x7fe3d8122500 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8123bf0 tx=0x7fe3c000e8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:47:05.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.979+0000 7fe3ceffd640 1 -- 172.21.15.62:0/804542104 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c0007450 con 0x7fe3d8122120 2024-04-19T01:47:05.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.979+0000 7fe3ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:47:05.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.979+0000 7fe3ceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:47:05.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.979+0000 7fe3ceffd640 10 monclient: dump: 2024-04-19T01:47:05.989 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:47:05.989 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:47:05.989 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:05.989 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:05.989 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:47:05.990 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:47:05.990 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:47:05.990 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:47:05.990 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:47:05.990 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:47:05.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.979+0000 7fe3ceffd640 1 -- 172.21.15.62:0/804542104 <== mon.0 v2:172.21.15.62:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe3c0018070 con 0x7fe3d8122120 2024-04-19T01:47:05.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.979+0000 7fe3ceffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:47:05.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.979+0000 7fe3ceffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:47:05.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.979+0000 7fe3ceffd640 1 -- 172.21.15.62:0/804542104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3d8122120 msgr2=0x7fe3d8122500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:47:05.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.979+0000 7fe3ceffd640 1 --2- 172.21.15.62:0/804542104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3d8122120 0x7fe3d8122500 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8123bf0 tx=0x7fe3c000e8b0 comp rx=0 tx=0).stop 2024-04-19T01:47:05.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.979+0000 7fe3ceffd640 1 -- 172.21.15.62:0/804542104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3d8122120 msgr2=0x7fe3d8122500 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:47:05.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.979+0000 7fe3ceffd640 1 --2- 172.21.15.62:0/804542104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3d8122120 0x7fe3d8122500 secure :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8123bf0 tx=0x7fe3c000e8b0 comp rx=0 tx=0).stop 2024-04-19T01:47:05.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.979+0000 7fe3d6d9d640 1 --2- 172.21.15.62:0/804542104 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fe3b8036450 0x7fe3b8038910 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0004ba0 tx=0x7fe3d0005090 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:47:05.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.987+0000 7fe3dca17640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:47:05.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.987+0000 7fe3dca17640 1 --2- 172.21.15.62:0/804542104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3d8122a40 0x7fe3d806fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:47:05.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.987+0000 7fe3dca17640 10 monclient(hunting): start opening mon connection 2024-04-19T01:47:05.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.987+0000 7fe3dca17640 1 -- 172.21.15.62:0/804542104 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe3d806b980 con 0x7fe3d8122a40 2024-04-19T01:47:05.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.987+0000 7fe3cffff640 1 --2- 172.21.15.62:0/804542104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3d8122a40 0x7fe3d806fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:47:05.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.987+0000 7fe3cffff640 10 monclient: get_auth_request con 0x7fe3d8122a40 auth_method 0 2024-04-19T01:47:05.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.987+0000 7fe3cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:47:05.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.987+0000 7fe3cffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:47:05.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.987+0000 7fe3cffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:47:05.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.987+0000 7fe3cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:47:05.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.987+0000 7fe3cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:47:05.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.987+0000 7fe3cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:47:05.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.987+0000 7fe3cffff640 10 monclient(hunting): handle_auth_done global_id 14835 payload 824 2024-04-19T01:47:05.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.987+0000 7fe3cffff640 1 -- 172.21.15.62:0/804542104 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe3d809df10 con 0x7fe3d8122a40 2024-04-19T01:47:05.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.987+0000 7fe3cffff640 1 -- 172.21.15.62:0/804542104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3d8122a40 msgr2=0x7fe3d806fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:47:05.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.987+0000 7fe3cffff640 1 --2- 172.21.15.62:0/804542104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3d8122a40 0x7fe3d806fb90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8123bf0 tx=0x7fe3c0003770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:47:05.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.987+0000 7fe3ceffd640 1 -- 172.21.15.62:0/804542104 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c001d9c0 con 0x7fe3d8122a40 2024-04-19T01:47:05.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.987+0000 7fe3ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:47:05.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.987+0000 7fe3ceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:47:05.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.987+0000 7fe3ceffd640 10 monclient: dump: 2024-04-19T01:47:05.992 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:47:05.992 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:47:05.992 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:05.992 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:05.992 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:47:05.992 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:47:05.992 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:47:05.992 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:47:05.992 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:47:05.992 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:47:05.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.987+0000 7fe3ceffd640 1 -- 172.21.15.62:0/804542104 <== mon.0 v2:172.21.15.62:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4708 (secure 0 0 0) 0x7fe3c001d9c0 con 0x7fe3d8122a40 2024-04-19T01:47:05.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.987+0000 7fe3ceffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:47:05.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.987+0000 7fe3ceffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:47:05.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.987+0000 7fe3ceffd640 1 -- 172.21.15.62:0/804542104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3d8122a40 msgr2=0x7fe3d806fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:47:05.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.987+0000 7fe3ceffd640 1 --2- 172.21.15.62:0/804542104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3d8122a40 0x7fe3d806fb90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8123bf0 tx=0x7fe3c0003770 comp rx=0 tx=0).stop 2024-04-19T01:47:05.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.987+0000 7fe3ceffd640 1 -- 172.21.15.62:0/804542104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3d8122a40 msgr2=0x7fe3d806fb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:47:05.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.987+0000 7fe3ceffd640 1 --2- 172.21.15.62:0/804542104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3d8122a40 0x7fe3d806fb90 secure :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8123bf0 tx=0x7fe3c0003770 comp rx=0 tx=0).stop 2024-04-19T01:47:05.993 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-19T01:47:05.993 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "a", 2024-04-19T01:47:05.993 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 0, 2024-04-19T01:47:05.993 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "leader", 2024-04-19T01:47:05.993 INFO:teuthology.orchestra.run.smithi062.stdout: "election_epoch": 38, 2024-04-19T01:47:05.993 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 169549, 2024-04-19T01:47:05.993 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum": [ 2024-04-19T01:47:05.993 INFO:teuthology.orchestra.run.smithi062.stdout: 0, 2024-04-19T01:47:05.993 INFO:teuthology.orchestra.run.smithi062.stdout: 1 2024-04-19T01:47:05.994 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:47:05.994 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_age": 0, 2024-04-19T01:47:05.994 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:47:05.994 INFO:teuthology.orchestra.run.smithi062.stdout: "required_con": "2450521708007866372", 2024-04-19T01:47:05.994 INFO:teuthology.orchestra.run.smithi062.stdout: "required_mon": [ 2024-04-19T01:47:05.994 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:47:05.994 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:47:05.994 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:47:05.994 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:47:05.994 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:47:05.994 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:47:05.994 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:47:05.994 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:47:05.994 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:47:05.995 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:47:05.995 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:47:05.995 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:47:05.995 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:47:05.995 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_mon": [ 2024-04-19T01:47:05.995 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:47:05.995 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:47:05.995 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:47:05.995 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:47:05.995 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:47:05.995 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:47:05.995 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:47:05.995 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:47:05.995 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:47:05.995 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:47:05.996 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:47:05.996 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:47:05.996 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:47:05.996 INFO:teuthology.orchestra.run.smithi062.stdout: "outside_quorum": [], 2024-04-19T01:47:05.996 INFO:teuthology.orchestra.run.smithi062.stdout: "extra_probe_peers": [], 2024-04-19T01:47:05.996 INFO:teuthology.orchestra.run.smithi062.stdout: "sync_provider": [], 2024-04-19T01:47:05.996 INFO:teuthology.orchestra.run.smithi062.stdout: "monmap": { 2024-04-19T01:47:05.996 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 1, 2024-04-19T01:47:05.996 INFO:teuthology.orchestra.run.smithi062.stdout: "fsid": "6466e4b6-2520-485f-95b3-9401915fbbad", 2024-04-19T01:47:05.996 INFO:teuthology.orchestra.run.smithi062.stdout: "modified": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:47:05.996 INFO:teuthology.orchestra.run.smithi062.stdout: "created": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:47:05.996 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release": 19, 2024-04-19T01:47:05.996 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release_name": "squid", 2024-04-19T01:47:05.996 INFO:teuthology.orchestra.run.smithi062.stdout: "election_strategy": 1, 2024-04-19T01:47:05.996 INFO:teuthology.orchestra.run.smithi062.stdout: "disallowed_leaders: ": "", 2024-04-19T01:47:05.996 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false, 2024-04-19T01:47:05.997 INFO:teuthology.orchestra.run.smithi062.stdout: "tiebreaker_mon": "", 2024-04-19T01:47:05.997 INFO:teuthology.orchestra.run.smithi062.stdout: "removed_ranks: ": "", 2024-04-19T01:47:05.997 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:47:05.997 INFO:teuthology.orchestra.run.smithi062.stdout: "persistent": [ 2024-04-19T01:47:05.997 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:47:05.997 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:47:05.997 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:47:05.997 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:47:05.997 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:47:05.997 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:47:05.997 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:47:05.997 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:47:05.997 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:47:05.997 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:47:05.997 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:47:05.998 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:47:05.998 INFO:teuthology.orchestra.run.smithi062.stdout: "optional": [] 2024-04-19T01:47:05.998 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:47:05.998 INFO:teuthology.orchestra.run.smithi062.stdout: "mons": [ 2024-04-19T01:47:05.998 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:47:05.998 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 0, 2024-04-19T01:47:05.998 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "a", 2024-04-19T01:47:05.998 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:47:05.998 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:47:05.998 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:47:05.998 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:47:05.998 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3300", 2024-04-19T01:47:05.998 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:47:05.998 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:47:05.999 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:47:05.999 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:47:05.999 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789", 2024-04-19T01:47:05.999 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:47:05.999 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:47:05.999 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:47:05.999 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:47:05.999 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789/0", 2024-04-19T01:47:05.999 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6789/0", 2024-04-19T01:47:05.999 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:47:05.999 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:47:05.999 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:47:05.999 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:47:05.999 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:47:06.000 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 1, 2024-04-19T01:47:06.000 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "b", 2024-04-19T01:47:06.000 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:47:06.000 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:47:06.000 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:47:06.000 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:47:06.000 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:3300", 2024-04-19T01:47:06.000 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:47:06.000 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:47:06.000 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:47:06.000 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:47:06.000 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789", 2024-04-19T01:47:06.000 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:47:06.000 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:47:06.000 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:47:06.001 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:47:06.001 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789/0", 2024-04-19T01:47:06.001 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.71:6789/0", 2024-04-19T01:47:06.001 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:47:06.001 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:47:06.001 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:47:06.001 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:47:06.001 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:47:06.001 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 2, 2024-04-19T01:47:06.001 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "c", 2024-04-19T01:47:06.001 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:47:06.001 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:47:06.001 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:47:06.001 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:47:06.002 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3301", 2024-04-19T01:47:06.002 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:47:06.002 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:47:06.002 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:47:06.002 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:47:06.002 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790", 2024-04-19T01:47:06.002 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:47:06.002 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:47:06.002 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:47:06.002 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:47:06.002 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790/0", 2024-04-19T01:47:06.002 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6790/0", 2024-04-19T01:47:06.002 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:47:06.002 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:47:06.002 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:47:06.002 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:47:06.003 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:47:06.003 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:47:06.003 INFO:teuthology.orchestra.run.smithi062.stdout: "feature_map": { 2024-04-19T01:47:06.003 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": [ 2024-04-19T01:47:06.003 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:47:06.003 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:47:06.003 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:47:06.003 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:47:06.003 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:47:06.003 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:47:06.003 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": [ 2024-04-19T01:47:06.003 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:47:06.003 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:47:06.003 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:47:06.003 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 5 2024-04-19T01:47:06.004 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:47:06.004 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:47:06.004 INFO:teuthology.orchestra.run.smithi062.stdout: "client": [ 2024-04-19T01:47:06.004 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:47:06.004 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:47:06.004 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:47:06.004 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 6 2024-04-19T01:47:06.004 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:47:06.004 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:47:06.004 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:47:06.004 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false 2024-04-19T01:47:06.004 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-19T01:47:06.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.991+0000 7fe3dca17640 1 -- 172.21.15.62:0/804542104 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fe3b8036450 msgr2=0x7fe3b8038910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:47:06.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.991+0000 7fe3dca17640 1 --2- 172.21.15.62:0/804542104 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fe3b8036450 0x7fe3b8038910 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0004ba0 tx=0x7fe3d0005090 comp rx=0 tx=0).stop 2024-04-19T01:47:06.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.991+0000 7fe3dca17640 10 monclient: shutdown 2024-04-19T01:47:06.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.991+0000 7fe3dca17640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:47:06.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.991+0000 7fe3dca17640 1 -- 172.21.15.62:0/804542104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3d809f890 msgr2=0x7fe3d81cf630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:47:06.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.991+0000 7fe3dca17640 1 --2- 172.21.15.62:0/804542104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3d809f890 0x7fe3d81cf630 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe3c801aaa0 tx=0x7fe3c8019190 comp rx=0 tx=0).stop 2024-04-19T01:47:06.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.991+0000 7fe3d6d9d640 1 -- 172.21.15.62:0/804542104 reap_dead start 2024-04-19T01:47:06.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.991+0000 7fe3d6d9d640 1 -- 172.21.15.62:0/804542104 reap_dead start 2024-04-19T01:47:06.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.991+0000 7fe3dca17640 1 -- 172.21.15.62:0/804542104 shutdown_connections 2024-04-19T01:47:06.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.991+0000 7fe3dca17640 1 -- 172.21.15.62:0/804542104 >> 172.21.15.62:0/804542104 conn(0x7fe3d80a23d0 msgr2=0x7fe3d813aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:47:06.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.991+0000 7fe3dca17640 1 -- 172.21.15.62:0/804542104 shutdown_connections 2024-04-19T01:47:06.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:05.991+0000 7fe3dca17640 1 -- 172.21.15.62:0/804542104 wait complete. 2024-04-19T01:47:06.007 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-19T01:47:06.007 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-04-19T01:47:06.007 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T01:47:06.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad8cdc2640 10 monclient: get_monmap_and_config 2024-04-19T01:47:06.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad8cdc2640 10 monclient: build_initial_monmap 2024-04-19T01:47:06.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad8cdc2640 10 monclient: monmap: 2024-04-19T01:47:06.125 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:47:06.125 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:47:06.125 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:47:06.130981+0000 2024-04-19T01:47:06.125 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:47:06.130981+0000 2024-04-19T01:47:06.125 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:47:06.126 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:47:06.126 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:47:06.126 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:47:06.126 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:47:06.126 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:47:06.126 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:47:06.126 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:47:06.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad8cdc2640 1 Processor -- start 2024-04-19T01:47:06.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad8cdc2640 1 -- start start 2024-04-19T01:47:06.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad8cdc2640 10 monclient: init 2024-04-19T01:47:06.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad8cdc2640 10 monclient: _reopen_session rank -1 2024-04-19T01:47:06.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad8cdc2640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-04-19T01:47:06.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad8cdc2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad880a2ef0 0x7fad881379a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:47:06.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad8cdc2640 10 monclient(hunting): picked mon.noname-b con 0x7fad880a2ef0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:47:06.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad8cdc2640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fad88137ee0 0x7fad880a47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:47:06.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad8cdc2640 10 monclient(hunting): picked mon.noname-c con 0x7fad88137ee0 addr v2:172.21.15.71:3300/0 2024-04-19T01:47:06.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad8cdc2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fad880a4d20 addr v1:172.21.15.71:6789/0 2024-04-19T01:47:06.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad8cdc2640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad88137550 con 0x7fad880a4d20 2024-04-19T01:47:06.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad8cdc2640 10 monclient(hunting): start opening mon connection 2024-04-19T01:47:06.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad8cdc2640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fad8809bd90 con 0x7fad88137ee0 2024-04-19T01:47:06.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad8cdc2640 10 monclient(hunting): start opening mon connection 2024-04-19T01:47:06.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad8cdc2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fad8809c070 con 0x7fad880a2ef0 2024-04-19T01:47:06.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad8cdc2640 10 monclient(hunting): _renew_subs 2024-04-19T01:47:06.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad8cdc2640 10 monclient(hunting): authenticate will time out at 1211.950488s 2024-04-19T01:47:06.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad8659c640 1 -- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad880a2ef0 msgr2=0x7fad881379a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.62:3301/0 2024-04-19T01:47:06.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad8659c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad880a2ef0 0x7fad881379a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:47:06.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad85d9b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fad88137ee0 0x7fad880a47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:47:06.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad85d9b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fad88137ee0 0x7fad880a47e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.62:42400/0 (socket says 172.21.15.62:42400) 2024-04-19T01:47:06.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad85d9b640 1 -- 172.21.15.62:0/416049408 learned_addr learned my addr 172.21.15.62:0/416049408 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:47:06.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad86d9d640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fad880a4d20 0x7fad8813e590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.62:54774/0 (socket says 172.21.15.62:54774) 2024-04-19T01:47:06.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad85d9b640 10 monclient(hunting): get_auth_request con 0x7fad88137ee0 auth_method 0 2024-04-19T01:47:06.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad85d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:47:06.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad85d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:47:06.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad85d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:47:06.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad85d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:47:06.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad85d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:47:06.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad85d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:47:06.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad85d9b640 10 monclient(hunting): handle_auth_done global_id 24151 payload 274 2024-04-19T01:47:06.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad85d9b640 1 -- 172.21.15.62:0/416049408 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad880a2ef0 msgr2=0x7fad881379a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:47:06.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad85d9b640 1 --2- 172.21.15.62:0/416049408 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad880a2ef0 0x7fad881379a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:47:06.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad85d9b640 1 -- 172.21.15.62:0/416049408 >> v1:172.21.15.71:6789/0 conn(0x7fad880a4d20 legacy=0x7fad8813e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:47:06.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad85d9b640 10 monclient: _finish_hunting 0 2024-04-19T01:47:06.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad85d9b640 1 monclient: found mon.noname-c 2024-04-19T01:47:06.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad85d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:47:06.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad85d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-04-19T01:47:06.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.127+0000 7fad85d9b640 1 -- 172.21.15.62:0/416049408 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad8809c350 con 0x7fad88137ee0 2024-04-19T01:47:06.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad85d9b640 1 -- 172.21.15.62:0/416049408 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad88137ee0 msgr2=0x7fad880a47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:47:06.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad85d9b640 1 --2- 172.21.15.62:0/416049408 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad88137ee0 0x7fad880a47e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fad70009870 tx=0x7fad7001c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b2e92049030dd1c server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:47:06.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad8559a640 1 -- 172.21.15.62:0/416049408 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad700b2070 con 0x7fad88137ee0 2024-04-19T01:47:06.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad8559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:47:06.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad8559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:47:06.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad8559a640 10 monclient: dump: 2024-04-19T01:47:06.134 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:47:06.134 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:47:06.134 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:06.134 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:06.134 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:47:06.134 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:47:06.134 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:47:06.134 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:47:06.134 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:47:06.134 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:47:06.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad8559a640 10 monclient: _finish_auth 0 2024-04-19T01:47:06.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad8559a640 10 monclient: _check_auth_tickets 2024-04-19T01:47:06.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad8559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:47:06.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad8559a640 1 -- 172.21.15.62:0/416049408 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad700a9070 con 0x7fad88137ee0 2024-04-19T01:47:06.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad8cdc2640 5 monclient: authenticate success, global_id 24151 2024-04-19T01:47:06.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad8cdc2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:47:06.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad8559a640 10 monclient: handle_config config(0 keys) 2024-04-19T01:47:06.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad8559a640 1 -- 172.21.15.62:0/416049408 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad700ad880 con 0x7fad88137ee0 2024-04-19T01:47:06.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad8cdc2640 10 monclient: get_monmap_and_config success 2024-04-19T01:47:06.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad8cdc2640 4 set_mon_vals no callback set 2024-04-19T01:47:06.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad8559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:47:06.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad8559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:47:06.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad8559a640 10 monclient: dump: 2024-04-19T01:47:06.135 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:47:06.135 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:47:06.135 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:06.135 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:06.136 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:47:06.136 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:47:06.136 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:47:06.136 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:47:06.136 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:47:06.136 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:47:06.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad8cdc2640 10 monclient: shutdown 2024-04-19T01:47:06.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad8cdc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:47:06.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad8cdc2640 1 -- 172.21.15.62:0/416049408 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad88137ee0 msgr2=0x7fad880a47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:47:06.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad8cdc2640 1 --2- 172.21.15.62:0/416049408 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad88137ee0 0x7fad880a47e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fad70009870 tx=0x7fad7001c4d0 comp rx=0 tx=0).stop 2024-04-19T01:47:06.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad8cdc2640 1 -- 172.21.15.62:0/416049408 shutdown_connections 2024-04-19T01:47:06.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad8cdc2640 1 --2- 172.21.15.62:0/416049408 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad88137ee0 0x7fad880a47e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:47:06.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad8cdc2640 1 --2- 172.21.15.62:0/416049408 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad880a2ef0 0x7fad881379a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:47:06.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad8cdc2640 1 -- 172.21.15.62:0/416049408 >> 172.21.15.62:0/416049408 conn(0x7fad880a23d0 msgr2=0x7fad880a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:47:06.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad8cdc2640 1 -- 172.21.15.62:0/416049408 shutdown_connections 2024-04-19T01:47:06.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad8cdc2640 1 -- 172.21.15.62:0/416049408 wait complete. 2024-04-19T01:47:06.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad8cdc2640 10 monclient: build_initial_monmap 2024-04-19T01:47:06.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad8cdc2640 10 monclient: monmap: 2024-04-19T01:47:06.143 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:47:06.143 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:47:06.143 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:47:06.143 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:47:06.143 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:47:06.143 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:47:06.143 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:47:06.143 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:47:06.143 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:47:06.143 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:47:06.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad8cdc2640 1 Processor -- start 2024-04-19T01:47:06.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad8cdc2640 1 -- start start 2024-04-19T01:47:06.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.135+0000 7fad8cdc2640 10 monclient: init 2024-04-19T01:47:06.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad8cdc2640 10 monclient: _reopen_session rank -1 2024-04-19T01:47:06.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad8cdc2640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-19T01:47:06.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad8cdc2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad880a2ef0 0x7fad8811c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:47:06.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad8cdc2640 10 monclient(hunting): picked mon.noname-c con 0x7fad880a2ef0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:47:06.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad8cdc2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad880a4d20 0x7fad8813c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:47:06.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad8cdc2640 10 monclient(hunting): picked mon.noname-b con 0x7fad880a4d20 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:47:06.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad8cdc2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad881330d0 0x7fad881cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:47:06.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad8cdc2640 10 monclient(hunting): picked mon.noname-a con 0x7fad881330d0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:47:06.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad8cdc2640 10 monclient(hunting): start opening mon connection 2024-04-19T01:47:06.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad8cdc2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fad881424d0 con 0x7fad881330d0 2024-04-19T01:47:06.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad8cdc2640 10 monclient(hunting): start opening mon connection 2024-04-19T01:47:06.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad8cdc2640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fad88072520 con 0x7fad880a4d20 2024-04-19T01:47:06.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad8cdc2640 10 monclient(hunting): start opening mon connection 2024-04-19T01:47:06.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad8cdc2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fad88142650 con 0x7fad880a2ef0 2024-04-19T01:47:06.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad8cdc2640 10 monclient(hunting): _renew_subs 2024-04-19T01:47:06.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad8cdc2640 10 monclient(hunting): authenticate will time out at 1211.960141s 2024-04-19T01:47:06.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad8659c640 1 -- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad880a2ef0 msgr2=0x7fad8811c2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:3301/0 2024-04-19T01:47:06.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad8659c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad880a2ef0 0x7fad8811c2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:47:06.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad86d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad881330d0 0x7fad881cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:47:06.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad86d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad881330d0 0x7fad881cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35590/0 (socket says 172.21.15.62:35590) 2024-04-19T01:47:06.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad86d9d640 1 -- 172.21.15.62:0/3156118481 learned_addr learned my addr 172.21.15.62:0/3156118481 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:47:06.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad86d9d640 10 monclient(hunting): get_auth_request con 0x7fad881330d0 auth_method 0 2024-04-19T01:47:06.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad86d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:47:06.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad86d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:47:06.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad86d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:47:06.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad86d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:47:06.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad86d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:47:06.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad86d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:47:06.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad85d9b640 1 --2- 172.21.15.62:0/3156118481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad880a4d20 0x7fad8813c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:47:06.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad85d9b640 10 monclient(hunting): get_auth_request con 0x7fad880a4d20 auth_method 0 2024-04-19T01:47:06.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad85d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:47:06.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad85d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:47:06.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad85d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:47:06.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad86d9d640 10 monclient(hunting): handle_auth_done global_id 14838 payload 824 2024-04-19T01:47:06.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad86d9d640 1 -- 172.21.15.62:0/3156118481 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad880a2ef0 msgr2=0x7fad8811c2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:47:06.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad86d9d640 1 --2- 172.21.15.62:0/3156118481 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fad880a2ef0 0x7fad8811c2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:47:06.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad86d9d640 1 -- 172.21.15.62:0/3156118481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad880a4d20 msgr2=0x7fad8813c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:47:06.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad86d9d640 1 --2- 172.21.15.62:0/3156118481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad880a4d20 0x7fad8813c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:47:06.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad86d9d640 10 monclient: _finish_hunting 0 2024-04-19T01:47:06.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad86d9d640 1 monclient: found mon.noname-a 2024-04-19T01:47:06.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad86d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:47:06.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad86d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:47:06.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad86d9d640 1 -- 172.21.15.62:0/3156118481 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad8813e460 con 0x7fad881330d0 2024-04-19T01:47:06.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad86d9d640 1 -- 172.21.15.62:0/3156118481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad881330d0 msgr2=0x7fad881cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:47:06.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad86d9d640 1 --2- 172.21.15.62:0/3156118481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad881330d0 0x7fad881cf7d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fad7800c3e0 tx=0x7fad7800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:47:06.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad7effd640 1 -- 172.21.15.62:0/3156118481 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad78017070 con 0x7fad881330d0 2024-04-19T01:47:06.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:47:06.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad7effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:47:06.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad7effd640 10 monclient: dump: 2024-04-19T01:47:06.147 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:47:06.147 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:47:06.147 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:06.147 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:06.147 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:47:06.147 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:47:06.147 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:47:06.147 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:47:06.147 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:47:06.147 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:47:06.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad7effd640 10 monclient: _finish_auth 0 2024-04-19T01:47:06.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad7effd640 10 monclient: _check_auth_tickets 2024-04-19T01:47:06.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:47:06.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad7effd640 1 -- 172.21.15.62:0/3156118481 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad78005800 con 0x7fad881330d0 2024-04-19T01:47:06.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad8cdc2640 5 monclient: authenticate success, global_id 14838 2024-04-19T01:47:06.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad7effd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:47:06.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad8cdc2640 10 monclient: _renew_subs 2024-04-19T01:47:06.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad7effd640 1 -- 172.21.15.62:0/3156118481 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad78005bf0 con 0x7fad881330d0 2024-04-19T01:47:06.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad8cdc2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:47:06.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad8cdc2640 1 -- 172.21.15.62:0/3156118481 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad8809f450 con 0x7fad881330d0 2024-04-19T01:47:06.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad84d99640 4 set_mon_vals no callback set 2024-04-19T01:47:06.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:47:06.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad7effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:47:06.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad7effd640 10 monclient: dump: 2024-04-19T01:47:06.148 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:47:06.148 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:47:06.149 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:06.149 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:06.149 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:47:06.149 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:47:06.149 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:47:06.149 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:47:06.149 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:47:06.149 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:47:06.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad8cdc2640 10 monclient: _renew_subs 2024-04-19T01:47:06.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad8cdc2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:47:06.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad8cdc2640 1 -- 172.21.15.62:0/3156118481 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad88133550 con 0x7fad881330d0 2024-04-19T01:47:06.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad8cdc2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:47:06.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad8cdc2640 1 --2- 172.21.15.62:0/3156118481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad881cfd10 0x7fad881d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:47:06.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad8cdc2640 10 monclient(hunting): start opening mon connection 2024-04-19T01:47:06.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.139+0000 7fad8cdc2640 1 -- 172.21.15.62:0/3156118481 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fad8813f3e0 con 0x7fad881cfd10 2024-04-19T01:47:06.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.143+0000 7fad7effd640 1 -- 172.21.15.62:0/3156118481 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fad7801a070 con 0x7fad881330d0 2024-04-19T01:47:06.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.143+0000 7fad8659c640 1 --2- 172.21.15.62:0/3156118481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad881cfd10 0x7fad881d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:47:06.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.143+0000 7fad8659c640 10 monclient: get_auth_request con 0x7fad881cfd10 auth_method 0 2024-04-19T01:47:06.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.143+0000 7fad8659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:47:06.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.143+0000 7fad8659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:47:06.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.143+0000 7fad8659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:47:06.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.143+0000 7fad8659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:47:06.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.143+0000 7fad8659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:47:06.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.143+0000 7fad8659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:47:06.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.143+0000 7fad7effd640 1 --2- 172.21.15.62:0/3156118481 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fad680363d0 0x7fad68038890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:47:06.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.143+0000 7fad7effd640 1 -- 172.21.15.62:0/3156118481 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(225..225 src has 115..225) ==== 4346+0+0 (secure 0 0 0) 0x7fad78048700 con 0x7fad881330d0 2024-04-19T01:47:06.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.143+0000 7fad8659c640 10 monclient(hunting): handle_auth_done global_id 24157 payload 824 2024-04-19T01:47:06.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.143+0000 7fad8659c640 1 -- 172.21.15.62:0/3156118481 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fad881d0630 con 0x7fad881cfd10 2024-04-19T01:47:06.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.143+0000 7fad85d9b640 1 --2- 172.21.15.62:0/3156118481 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fad680363d0 0x7fad68038890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:47:06.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.143+0000 7fad85d9b640 10 monclient: get_auth_request con 0x7fad680363d0 auth_method 0 2024-04-19T01:47:06.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.143+0000 7fad8659c640 1 -- 172.21.15.62:0/3156118481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad881cfd10 msgr2=0x7fad881d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:47:06.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.143+0000 7fad8659c640 1 --2- 172.21.15.62:0/3156118481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad881cfd10 0x7fad881d00f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fad8000bcc0 tx=0x7fad80009f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:47:06.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.143+0000 7fad7effd640 1 -- 172.21.15.62:0/3156118481 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad80005560 con 0x7fad881cfd10 2024-04-19T01:47:06.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.143+0000 7fad7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:47:06.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.143+0000 7fad7effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:47:06.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.143+0000 7fad7effd640 10 monclient: dump: 2024-04-19T01:47:06.153 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:47:06.153 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:47:06.153 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:06.153 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:06.153 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:47:06.153 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:47:06.153 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:47:06.153 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:47:06.154 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:47:06.154 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:47:06.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.143+0000 7fad85d9b640 1 --2- 172.21.15.62:0/3156118481 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fad680363d0 0x7fad68038890 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fad700ac000 tx=0x7fad70008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:47:06.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.147+0000 7fad7effd640 1 -- 172.21.15.62:0/3156118481 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fad80005560 con 0x7fad881cfd10 2024-04-19T01:47:06.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.147+0000 7fad7effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:47:06.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.147+0000 7fad7effd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:47:06.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.147+0000 7fad7effd640 1 -- 172.21.15.62:0/3156118481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad881cfd10 msgr2=0x7fad881d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:47:06.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.147+0000 7fad7effd640 1 --2- 172.21.15.62:0/3156118481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad881cfd10 0x7fad881d00f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fad8000bcc0 tx=0x7fad80009f30 comp rx=0 tx=0).stop 2024-04-19T01:47:06.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.147+0000 7fad7effd640 1 -- 172.21.15.62:0/3156118481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad881cfd10 msgr2=0x7fad881d00f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:47:06.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.147+0000 7fad7effd640 1 --2- 172.21.15.62:0/3156118481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad881cfd10 0x7fad881d00f0 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fad8000bcc0 tx=0x7fad80009f30 comp rx=0 tx=0).stop 2024-04-19T01:47:06.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.151+0000 7fad8cdc2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:47:06.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.151+0000 7fad8cdc2640 1 --2- 172.21.15.62:0/3156118481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad881382c0 0x7fad881386d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:47:06.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.151+0000 7fad8cdc2640 10 monclient(hunting): start opening mon connection 2024-04-19T01:47:06.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.151+0000 7fad8cdc2640 1 -- 172.21.15.62:0/3156118481 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fad881337b0 con 0x7fad881382c0 2024-04-19T01:47:06.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.151+0000 7fad8659c640 1 --2- 172.21.15.62:0/3156118481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad881382c0 0x7fad881386d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:47:06.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.151+0000 7fad8659c640 10 monclient: get_auth_request con 0x7fad881382c0 auth_method 0 2024-04-19T01:47:06.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.151+0000 7fad8659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:47:06.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.151+0000 7fad8659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:47:06.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.151+0000 7fad8659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:47:06.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.151+0000 7fad8659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:47:06.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.151+0000 7fad8659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:47:06.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.151+0000 7fad8659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:47:06.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.151+0000 7fad8659c640 10 monclient(hunting): handle_auth_done global_id 24160 payload 824 2024-04-19T01:47:06.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.151+0000 7fad8659c640 1 -- 172.21.15.62:0/3156118481 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fad880692b0 con 0x7fad881382c0 2024-04-19T01:47:06.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.155+0000 7fad8659c640 1 -- 172.21.15.62:0/3156118481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad881382c0 msgr2=0x7fad881386d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:47:06.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.155+0000 7fad8659c640 1 --2- 172.21.15.62:0/3156118481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad881382c0 0x7fad881386d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fad80014000 tx=0x7fad80005dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:47:06.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.155+0000 7fad7effd640 1 -- 172.21.15.62:0/3156118481 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad80019040 con 0x7fad881382c0 2024-04-19T01:47:06.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.155+0000 7fad7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:47:06.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.155+0000 7fad7effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:47:06.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.155+0000 7fad7effd640 10 monclient: dump: 2024-04-19T01:47:06.157 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:47:06.157 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:47:06.157 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:06.157 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:47:06.157 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:47:06.157 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:47:06.157 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:47:06.157 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:47:06.157 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:47:06.157 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:47:06.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.155+0000 7fad7effd640 1 -- 172.21.15.62:0/3156118481 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7fad8001ac30 con 0x7fad881382c0 2024-04-19T01:47:06.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.155+0000 7fad7effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:47:06.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.155+0000 7fad7effd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:47:06.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.155+0000 7fad7effd640 1 -- 172.21.15.62:0/3156118481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad881382c0 msgr2=0x7fad881386d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:47:06.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.155+0000 7fad7effd640 1 --2- 172.21.15.62:0/3156118481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad881382c0 0x7fad881386d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fad80014000 tx=0x7fad80005dd0 comp rx=0 tx=0).stop 2024-04-19T01:47:06.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.155+0000 7fad7effd640 1 -- 172.21.15.62:0/3156118481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad881382c0 msgr2=0x7fad881386d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:47:06.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.155+0000 7fad7effd640 1 --2- 172.21.15.62:0/3156118481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad881382c0 0x7fad881386d0 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fad80014000 tx=0x7fad80005dd0 comp rx=0 tx=0).stop 2024-04-19T01:47:06.158 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-19T01:47:06.158 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "b", 2024-04-19T01:47:06.158 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 1, 2024-04-19T01:47:06.158 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "peon", 2024-04-19T01:47:06.158 INFO:teuthology.orchestra.run.smithi062.stdout: "election_epoch": 38, 2024-04-19T01:47:06.158 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 32803, 2024-04-19T01:47:06.158 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum": [ 2024-04-19T01:47:06.159 INFO:teuthology.orchestra.run.smithi062.stdout: 0, 2024-04-19T01:47:06.159 INFO:teuthology.orchestra.run.smithi062.stdout: 1 2024-04-19T01:47:06.159 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:47:06.159 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_age": 0, 2024-04-19T01:47:06.159 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:47:06.159 INFO:teuthology.orchestra.run.smithi062.stdout: "required_con": "2450521708007866372", 2024-04-19T01:47:06.159 INFO:teuthology.orchestra.run.smithi062.stdout: "required_mon": [ 2024-04-19T01:47:06.159 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:47:06.159 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:47:06.159 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:47:06.159 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:47:06.159 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:47:06.159 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:47:06.159 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:47:06.159 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:47:06.160 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:47:06.160 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:47:06.160 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:47:06.160 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:47:06.160 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:47:06.160 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_mon": [ 2024-04-19T01:47:06.160 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:47:06.160 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:47:06.160 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:47:06.160 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:47:06.160 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:47:06.160 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:47:06.160 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:47:06.160 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:47:06.160 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:47:06.161 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:47:06.161 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:47:06.161 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:47:06.161 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:47:06.161 INFO:teuthology.orchestra.run.smithi062.stdout: "outside_quorum": [], 2024-04-19T01:47:06.161 INFO:teuthology.orchestra.run.smithi062.stdout: "extra_probe_peers": [], 2024-04-19T01:47:06.161 INFO:teuthology.orchestra.run.smithi062.stdout: "sync_provider": [], 2024-04-19T01:47:06.161 INFO:teuthology.orchestra.run.smithi062.stdout: "monmap": { 2024-04-19T01:47:06.161 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 1, 2024-04-19T01:47:06.161 INFO:teuthology.orchestra.run.smithi062.stdout: "fsid": "6466e4b6-2520-485f-95b3-9401915fbbad", 2024-04-19T01:47:06.161 INFO:teuthology.orchestra.run.smithi062.stdout: "modified": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:47:06.161 INFO:teuthology.orchestra.run.smithi062.stdout: "created": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:47:06.161 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release": 19, 2024-04-19T01:47:06.161 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release_name": "squid", 2024-04-19T01:47:06.162 INFO:teuthology.orchestra.run.smithi062.stdout: "election_strategy": 1, 2024-04-19T01:47:06.162 INFO:teuthology.orchestra.run.smithi062.stdout: "disallowed_leaders: ": "", 2024-04-19T01:47:06.162 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false, 2024-04-19T01:47:06.162 INFO:teuthology.orchestra.run.smithi062.stdout: "tiebreaker_mon": "", 2024-04-19T01:47:06.162 INFO:teuthology.orchestra.run.smithi062.stdout: "removed_ranks: ": "", 2024-04-19T01:47:06.162 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:47:06.162 INFO:teuthology.orchestra.run.smithi062.stdout: "persistent": [ 2024-04-19T01:47:06.162 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:47:06.162 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:47:06.162 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:47:06.162 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:47:06.162 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:47:06.162 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:47:06.162 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:47:06.162 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:47:06.163 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:47:06.163 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:47:06.163 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:47:06.163 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:47:06.163 INFO:teuthology.orchestra.run.smithi062.stdout: "optional": [] 2024-04-19T01:47:06.163 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:47:06.163 INFO:teuthology.orchestra.run.smithi062.stdout: "mons": [ 2024-04-19T01:47:06.163 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:47:06.163 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 0, 2024-04-19T01:47:06.163 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "a", 2024-04-19T01:47:06.163 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:47:06.163 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:47:06.163 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:47:06.163 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:47:06.164 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3300", 2024-04-19T01:47:06.164 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:47:06.164 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:47:06.164 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:47:06.164 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:47:06.164 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789", 2024-04-19T01:47:06.164 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:47:06.164 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:47:06.164 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:47:06.164 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:47:06.164 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789/0", 2024-04-19T01:47:06.164 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6789/0", 2024-04-19T01:47:06.164 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:47:06.164 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:47:06.165 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:47:06.165 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:47:06.165 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:47:06.165 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 1, 2024-04-19T01:47:06.165 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "b", 2024-04-19T01:47:06.165 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:47:06.165 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:47:06.165 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:47:06.165 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:47:06.165 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:3300", 2024-04-19T01:47:06.165 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:47:06.165 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:47:06.165 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:47:06.165 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:47:06.165 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789", 2024-04-19T01:47:06.166 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:47:06.166 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:47:06.166 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:47:06.166 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:47:06.166 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789/0", 2024-04-19T01:47:06.166 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.71:6789/0", 2024-04-19T01:47:06.166 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:47:06.166 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:47:06.166 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:47:06.166 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:47:06.166 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:47:06.166 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 2, 2024-04-19T01:47:06.166 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "c", 2024-04-19T01:47:06.166 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:47:06.167 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:47:06.167 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:47:06.167 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:47:06.167 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3301", 2024-04-19T01:47:06.167 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:47:06.167 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:47:06.167 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:47:06.167 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:47:06.167 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790", 2024-04-19T01:47:06.167 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:47:06.167 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:47:06.167 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:47:06.167 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:47:06.167 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790/0", 2024-04-19T01:47:06.167 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6790/0", 2024-04-19T01:47:06.168 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:47:06.168 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:47:06.168 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:47:06.168 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:47:06.168 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:47:06.168 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:47:06.168 INFO:teuthology.orchestra.run.smithi062.stdout: "feature_map": { 2024-04-19T01:47:06.168 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": [ 2024-04-19T01:47:06.168 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:47:06.168 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:47:06.168 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:47:06.168 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:47:06.168 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:47:06.168 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:47:06.168 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": [ 2024-04-19T01:47:06.169 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:47:06.169 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:47:06.169 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:47:06.169 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:47:06.169 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:47:06.169 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:47:06.169 INFO:teuthology.orchestra.run.smithi062.stdout: "client": [ 2024-04-19T01:47:06.169 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:47:06.169 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:47:06.169 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:47:06.169 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 4 2024-04-19T01:47:06.169 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:47:06.169 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:47:06.169 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": [ 2024-04-19T01:47:06.169 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:47:06.170 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:47:06.170 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:47:06.170 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:47:06.170 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:47:06.170 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:47:06.170 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:47:06.170 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false 2024-04-19T01:47:06.170 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-19T01:47:06.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.155+0000 7fad8cdc2640 1 -- 172.21.15.62:0/3156118481 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fad680363d0 msgr2=0x7fad68038890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:47:06.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.155+0000 7fad8cdc2640 1 --2- 172.21.15.62:0/3156118481 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fad680363d0 0x7fad68038890 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fad700ac000 tx=0x7fad70008040 comp rx=0 tx=0).stop 2024-04-19T01:47:06.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.155+0000 7fad8cdc2640 10 monclient: shutdown 2024-04-19T01:47:06.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.155+0000 7fad8cdc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:47:06.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.155+0000 7fad8cdc2640 1 -- 172.21.15.62:0/3156118481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad881330d0 msgr2=0x7fad881cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:47:06.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.155+0000 7fad8cdc2640 1 --2- 172.21.15.62:0/3156118481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad881330d0 0x7fad881cf7d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fad7800c3e0 tx=0x7fad7800c8a0 comp rx=0 tx=0).stop 2024-04-19T01:47:06.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.155+0000 7fad86d9d640 1 -- 172.21.15.62:0/3156118481 reap_dead start 2024-04-19T01:47:06.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.155+0000 7fad86d9d640 1 -- 172.21.15.62:0/3156118481 reap_dead start 2024-04-19T01:47:06.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.155+0000 7fad8cdc2640 1 -- 172.21.15.62:0/3156118481 shutdown_connections 2024-04-19T01:47:06.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.155+0000 7fad8cdc2640 1 -- 172.21.15.62:0/3156118481 >> 172.21.15.62:0/3156118481 conn(0x7fad880a23d0 msgr2=0x7fad881362c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:47:06.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.155+0000 7fad8cdc2640 1 -- 172.21.15.62:0/3156118481 shutdown_connections 2024-04-19T01:47:06.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:47:06.155+0000 7fad8cdc2640 1 -- 172.21.15.62:0/3156118481 wait complete. 2024-04-19T01:47:06.173 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-19T01:47:06.173 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-04-19T01:47:07.788 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:07.789 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30519]: starting. 2024-04-19T01:47:07.789 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30519]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-30484 2024-04-19T01:47:07.789 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30519]: created object 0... 2024-04-19T01:47:07.789 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30519]: created object 25... 2024-04-19T01:47:07.789 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30519]: created object 49... 2024-04-19T01:47:07.789 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30519]: finishing. 2024-04-19T01:47:07.789 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30519]: shutting down. 2024-04-19T01:47:07.850 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:07.850 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30521]: starting. 2024-04-19T01:47:07.850 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30521]: listing objects. 2024-04-19T01:47:07.850 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30521]: listed object 0... 2024-04-19T01:47:07.850 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30521]: listed object 25... 2024-04-19T01:47:07.850 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30521]: saw 50 objects 2024-04-19T01:47:07.851 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30521]: shutting down. 2024-04-19T01:47:08.576 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:08.577 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30520]: starting. 2024-04-19T01:47:08.577 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30520]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-30484 2024-04-19T01:47:08.577 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30520]: shutting down. 2024-04-19T01:47:08.579 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:08.579 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:08.579 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:47:08.584 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:47:10.820 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30573]: starting. 2024-04-19T01:47:10.820 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30573]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-30572 2024-04-19T01:47:10.820 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30573]: created object 0... 2024-04-19T01:47:10.820 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30573]: created object 25... 2024-04-19T01:47:10.821 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30573]: created object 49... 2024-04-19T01:47:10.821 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30573]: finishing. 2024-04-19T01:47:10.821 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30573]: shutting down. 2024-04-19T01:47:11.500 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30574]: starting. 2024-04-19T01:47:11.500 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30574]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-30572 2024-04-19T01:47:11.500 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30574]: shutting down. 2024-04-19T01:47:13.730 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:13.730 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30607]: starting. 2024-04-19T01:47:13.730 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30607]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-30572 2024-04-19T01:47:13.730 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30607]: created object 0... 2024-04-19T01:47:13.730 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30607]: created object 25... 2024-04-19T01:47:13.730 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30607]: created object 49... 2024-04-19T01:47:13.730 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30607]: finishing. 2024-04-19T01:47:13.731 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30607]: shutting down. 2024-04-19T01:47:13.790 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:13.791 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30609]: starting. 2024-04-19T01:47:13.791 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30609]: listing objects. 2024-04-19T01:47:13.791 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30609]: listed object 0... 2024-04-19T01:47:13.791 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30609]: listed object 25... 2024-04-19T01:47:13.791 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30609]: saw 50 objects 2024-04-19T01:47:13.791 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30609]: shutting down. 2024-04-19T01:47:14.536 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:14.537 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30608]: starting. 2024-04-19T01:47:14.537 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30608]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-30572 2024-04-19T01:47:14.537 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30608]: shutting down. 2024-04-19T01:47:14.540 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:14.540 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:14.540 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:47:14.543 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:47:16.780 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30660]: starting. 2024-04-19T01:47:16.780 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30660]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-30659 2024-04-19T01:47:16.781 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30660]: created object 0... 2024-04-19T01:47:16.781 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30660]: created object 25... 2024-04-19T01:47:16.781 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30660]: created object 49... 2024-04-19T01:47:16.781 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30660]: finishing. 2024-04-19T01:47:16.781 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30660]: shutting down. 2024-04-19T01:47:17.550 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30661]: starting. 2024-04-19T01:47:17.550 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30661]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-30659 2024-04-19T01:47:17.550 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30661]: shutting down. 2024-04-19T01:47:19.790 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:19.790 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30694]: starting. 2024-04-19T01:47:19.790 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30694]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-30659 2024-04-19T01:47:19.790 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30694]: created object 0... 2024-04-19T01:47:19.790 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30694]: created object 25... 2024-04-19T01:47:19.790 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30694]: created object 49... 2024-04-19T01:47:19.790 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30694]: finishing. 2024-04-19T01:47:19.790 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30694]: shutting down. 2024-04-19T01:47:20.698 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:20.698 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30696]: starting. 2024-04-19T01:47:20.698 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30696]: listing objects. 2024-04-19T01:47:20.698 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30696]: listed object 0... 2024-04-19T01:47:20.698 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30696]: listed object 25... 2024-04-19T01:47:20.699 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30696]: saw 50 objects 2024-04-19T01:47:20.699 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30696]: shutting down. 2024-04-19T01:47:21.508 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:21.508 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30695]: starting. 2024-04-19T01:47:21.508 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30695]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-30659 2024-04-19T01:47:21.508 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30695]: shutting down. 2024-04-19T01:47:21.510 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:21.511 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:21.511 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:47:21.514 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:47:23.738 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30747]: starting. 2024-04-19T01:47:23.739 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30747]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-30746 2024-04-19T01:47:23.739 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30747]: created object 0... 2024-04-19T01:47:23.739 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30747]: created object 25... 2024-04-19T01:47:23.739 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30747]: created object 49... 2024-04-19T01:47:23.739 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30747]: finishing. 2024-04-19T01:47:23.739 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30747]: shutting down. 2024-04-19T01:47:24.533 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30748]: starting. 2024-04-19T01:47:24.533 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30748]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-30746 2024-04-19T01:47:24.533 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30748]: shutting down. 2024-04-19T01:47:26.819 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:26.819 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30781]: starting. 2024-04-19T01:47:26.819 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30781]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-30746 2024-04-19T01:47:26.819 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30781]: created object 0... 2024-04-19T01:47:26.820 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30781]: created object 25... 2024-04-19T01:47:26.820 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30781]: created object 49... 2024-04-19T01:47:26.820 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30781]: finishing. 2024-04-19T01:47:26.820 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30781]: shutting down. 2024-04-19T01:47:26.880 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:26.880 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30783]: starting. 2024-04-19T01:47:26.880 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30783]: listing objects. 2024-04-19T01:47:26.880 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30783]: listed object 0... 2024-04-19T01:47:26.880 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30783]: listed object 25... 2024-04-19T01:47:26.880 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30783]: saw 50 objects 2024-04-19T01:47:26.880 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30783]: shutting down. 2024-04-19T01:47:27.626 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:27.626 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30782]: starting. 2024-04-19T01:47:27.626 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30782]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-30746 2024-04-19T01:47:27.626 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30782]: shutting down. 2024-04-19T01:47:27.630 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:27.630 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:27.630 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:47:27.634 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:47:29.874 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30834]: starting. 2024-04-19T01:47:29.874 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30834]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-30833 2024-04-19T01:47:29.874 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30834]: created object 0... 2024-04-19T01:47:29.874 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30834]: created object 25... 2024-04-19T01:47:29.874 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30834]: created object 49... 2024-04-19T01:47:29.874 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30834]: finishing. 2024-04-19T01:47:29.874 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30834]: shutting down. 2024-04-19T01:47:30.646 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30835]: starting. 2024-04-19T01:47:30.646 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30835]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-30833 2024-04-19T01:47:30.647 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30835]: shutting down. 2024-04-19T01:47:32.725 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:32.725 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30868]: starting. 2024-04-19T01:47:32.725 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30868]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-30833 2024-04-19T01:47:32.725 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30868]: created object 0... 2024-04-19T01:47:32.725 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30868]: created object 25... 2024-04-19T01:47:32.726 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30868]: created object 49... 2024-04-19T01:47:32.726 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30868]: finishing. 2024-04-19T01:47:32.726 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30868]: shutting down. 2024-04-19T01:47:32.784 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:32.785 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30870]: starting. 2024-04-19T01:47:32.785 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30870]: listing objects. 2024-04-19T01:47:32.785 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30870]: listed object 0... 2024-04-19T01:47:32.785 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30870]: listed object 25... 2024-04-19T01:47:32.785 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30870]: saw 50 objects 2024-04-19T01:47:32.785 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30870]: shutting down. 2024-04-19T01:47:33.515 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:33.515 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30869]: starting. 2024-04-19T01:47:33.515 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30869]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-30833 2024-04-19T01:47:33.516 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30869]: shutting down. 2024-04-19T01:47:33.518 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:33.518 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:33.518 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:47:33.521 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:47:35.739 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30921]: starting. 2024-04-19T01:47:35.739 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30921]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-30920 2024-04-19T01:47:35.739 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30921]: created object 0... 2024-04-19T01:47:35.739 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30921]: created object 25... 2024-04-19T01:47:35.739 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30921]: created object 49... 2024-04-19T01:47:35.739 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30921]: finishing. 2024-04-19T01:47:35.739 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[30921]: shutting down. 2024-04-19T01:47:36.525 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30922]: starting. 2024-04-19T01:47:36.525 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30922]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-30920 2024-04-19T01:47:36.526 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[30922]: shutting down. 2024-04-19T01:47:38.760 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:38.760 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30955]: starting. 2024-04-19T01:47:38.760 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30955]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-30920 2024-04-19T01:47:38.760 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30955]: created object 0... 2024-04-19T01:47:38.760 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30955]: created object 25... 2024-04-19T01:47:38.760 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30955]: created object 49... 2024-04-19T01:47:38.760 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30955]: finishing. 2024-04-19T01:47:38.760 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[30955]: shutting down. 2024-04-19T01:47:38.818 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:38.818 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30957]: starting. 2024-04-19T01:47:38.818 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30957]: listing objects. 2024-04-19T01:47:38.818 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30957]: listed object 0... 2024-04-19T01:47:38.818 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30957]: listed object 25... 2024-04-19T01:47:38.818 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30957]: saw 50 objects 2024-04-19T01:47:38.818 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[30957]: shutting down. 2024-04-19T01:47:39.554 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:39.555 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30956]: starting. 2024-04-19T01:47:39.555 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30956]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-30920 2024-04-19T01:47:39.555 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[30956]: shutting down. 2024-04-19T01:47:39.557 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:39.557 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:39.557 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:47:39.561 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:47:41.844 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31008]: starting. 2024-04-19T01:47:41.844 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31008]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-31007 2024-04-19T01:47:41.844 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31008]: created object 0... 2024-04-19T01:47:41.845 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31008]: created object 25... 2024-04-19T01:47:41.845 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31008]: created object 49... 2024-04-19T01:47:41.845 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31008]: finishing. 2024-04-19T01:47:41.845 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31008]: shutting down. 2024-04-19T01:47:43.651 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[31009]: starting. 2024-04-19T01:47:43.651 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[31009]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-31007 2024-04-19T01:47:43.651 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[31009]: shutting down. 2024-04-19T01:47:45.897 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:45.897 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31042]: starting. 2024-04-19T01:47:45.897 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31042]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-31007 2024-04-19T01:47:45.897 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31042]: created object 0... 2024-04-19T01:47:45.897 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31042]: created object 25... 2024-04-19T01:47:45.897 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31042]: created object 49... 2024-04-19T01:47:45.897 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31042]: finishing. 2024-04-19T01:47:45.898 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31042]: shutting down. 2024-04-19T01:47:45.955 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:45.955 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31044]: starting. 2024-04-19T01:47:45.955 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31044]: listing objects. 2024-04-19T01:47:45.955 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31044]: listed object 0... 2024-04-19T01:47:45.955 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31044]: listed object 25... 2024-04-19T01:47:45.955 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31044]: saw 50 objects 2024-04-19T01:47:45.955 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31044]: shutting down. 2024-04-19T01:47:46.685 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:46.685 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[31043]: starting. 2024-04-19T01:47:46.685 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[31043]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-31007 2024-04-19T01:47:46.685 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[31043]: shutting down. 2024-04-19T01:47:46.689 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:46.689 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:46.689 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:47:46.691 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:47:48.912 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31095]: starting. 2024-04-19T01:47:48.912 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31095]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-31094 2024-04-19T01:47:48.912 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31095]: created object 0... 2024-04-19T01:47:48.912 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31095]: created object 25... 2024-04-19T01:47:48.912 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31095]: created object 49... 2024-04-19T01:47:48.912 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31095]: finishing. 2024-04-19T01:47:48.913 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31095]: shutting down. 2024-04-19T01:47:49.721 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[31096]: starting. 2024-04-19T01:47:49.722 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[31096]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-31094 2024-04-19T01:47:49.722 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[31096]: shutting down. 2024-04-19T01:47:51.738 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:51.738 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31129]: starting. 2024-04-19T01:47:51.738 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31129]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-31094 2024-04-19T01:47:51.738 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31129]: created object 0... 2024-04-19T01:47:51.739 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31129]: created object 25... 2024-04-19T01:47:51.739 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31129]: created object 49... 2024-04-19T01:47:51.739 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31129]: finishing. 2024-04-19T01:47:51.739 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31129]: shutting down. 2024-04-19T01:47:51.801 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:51.801 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31131]: starting. 2024-04-19T01:47:51.801 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31131]: listing objects. 2024-04-19T01:47:51.802 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31131]: listed object 0... 2024-04-19T01:47:51.802 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31131]: listed object 25... 2024-04-19T01:47:51.802 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31131]: saw 50 objects 2024-04-19T01:47:51.802 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31131]: shutting down. 2024-04-19T01:47:52.531 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:52.531 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[31130]: starting. 2024-04-19T01:47:52.531 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[31130]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-31094 2024-04-19T01:47:52.531 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[31130]: shutting down. 2024-04-19T01:47:52.535 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:52.535 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:52.535 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:47:52.540 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:47:54.734 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31182]: starting. 2024-04-19T01:47:54.735 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31182]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-31181 2024-04-19T01:47:54.735 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31182]: created object 0... 2024-04-19T01:47:54.735 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31182]: created object 25... 2024-04-19T01:47:54.735 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31182]: created object 49... 2024-04-19T01:47:54.735 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31182]: finishing. 2024-04-19T01:47:54.735 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31182]: shutting down. 2024-04-19T01:47:56.559 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[31183]: starting. 2024-04-19T01:47:56.559 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[31183]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-31181 2024-04-19T01:47:56.559 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[31183]: shutting down. 2024-04-19T01:47:58.804 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:58.805 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31216]: starting. 2024-04-19T01:47:58.805 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31216]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-31181 2024-04-19T01:47:58.805 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31216]: created object 0... 2024-04-19T01:47:58.805 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31216]: created object 25... 2024-04-19T01:47:58.805 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31216]: created object 49... 2024-04-19T01:47:58.805 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31216]: finishing. 2024-04-19T01:47:58.805 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31216]: shutting down. 2024-04-19T01:47:58.859 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:58.859 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31218]: starting. 2024-04-19T01:47:58.859 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31218]: listing objects. 2024-04-19T01:47:58.859 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31218]: listed object 0... 2024-04-19T01:47:58.859 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31218]: listed object 25... 2024-04-19T01:47:58.859 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31218]: saw 50 objects 2024-04-19T01:47:58.859 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31218]: shutting down. 2024-04-19T01:47:59.603 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:59.604 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[31217]: starting. 2024-04-19T01:47:59.604 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[31217]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-31181 2024-04-19T01:47:59.604 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[31217]: shutting down. 2024-04-19T01:47:59.607 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:59.607 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:47:59.607 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:47:59.610 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:48:01.726 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31269]: starting. 2024-04-19T01:48:01.726 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31269]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-31268 2024-04-19T01:48:01.726 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31269]: created object 0... 2024-04-19T01:48:01.727 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31269]: created object 25... 2024-04-19T01:48:01.727 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31269]: created object 49... 2024-04-19T01:48:01.727 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31269]: finishing. 2024-04-19T01:48:01.727 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31269]: shutting down. 2024-04-19T01:48:02.521 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[31270]: starting. 2024-04-19T01:48:02.521 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[31270]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-31268 2024-04-19T01:48:02.521 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[31270]: shutting down. 2024-04-19T01:48:04.777 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:04.778 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31303]: starting. 2024-04-19T01:48:04.778 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31303]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-31268 2024-04-19T01:48:04.778 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31303]: created object 0... 2024-04-19T01:48:04.778 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31303]: created object 25... 2024-04-19T01:48:04.778 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31303]: created object 49... 2024-04-19T01:48:04.778 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31303]: finishing. 2024-04-19T01:48:04.778 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31303]: shutting down. 2024-04-19T01:48:04.839 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:04.839 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31305]: starting. 2024-04-19T01:48:04.839 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31305]: listing objects. 2024-04-19T01:48:04.839 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31305]: listed object 0... 2024-04-19T01:48:04.839 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31305]: listed object 25... 2024-04-19T01:48:04.839 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31305]: saw 50 objects 2024-04-19T01:48:04.840 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31305]: shutting down. 2024-04-19T01:48:06.590 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:06.590 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[31304]: starting. 2024-04-19T01:48:06.590 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[31304]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-31268 2024-04-19T01:48:06.590 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[31304]: shutting down. 2024-04-19T01:48:06.593 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:06.593 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:06.593 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:48:06.596 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:48:08.807 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31356]: starting. 2024-04-19T01:48:08.807 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31356]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-31355 2024-04-19T01:48:08.807 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31356]: created object 0... 2024-04-19T01:48:08.807 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31356]: created object 25... 2024-04-19T01:48:08.807 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31356]: created object 49... 2024-04-19T01:48:08.808 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31356]: finishing. 2024-04-19T01:48:08.808 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31356]: shutting down. 2024-04-19T01:48:09.621 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[31357]: starting. 2024-04-19T01:48:09.621 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[31357]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-31355 2024-04-19T01:48:09.621 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[31357]: shutting down. 2024-04-19T01:48:11.734 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:11.734 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31390]: starting. 2024-04-19T01:48:11.734 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31390]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-31355 2024-04-19T01:48:11.734 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31390]: created object 0... 2024-04-19T01:48:11.735 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31390]: created object 25... 2024-04-19T01:48:11.735 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31390]: created object 49... 2024-04-19T01:48:11.735 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31390]: finishing. 2024-04-19T01:48:11.735 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31390]: shutting down. 2024-04-19T01:48:11.795 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:11.795 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31392]: starting. 2024-04-19T01:48:11.795 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31392]: listing objects. 2024-04-19T01:48:11.795 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31392]: listed object 0... 2024-04-19T01:48:11.795 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31392]: listed object 25... 2024-04-19T01:48:11.795 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31392]: saw 50 objects 2024-04-19T01:48:11.795 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31392]: shutting down. 2024-04-19T01:48:13.544 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:13.545 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[31391]: starting. 2024-04-19T01:48:13.545 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[31391]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-31355 2024-04-19T01:48:13.545 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[31391]: shutting down. 2024-04-19T01:48:13.547 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:13.548 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:13.548 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:48:13.551 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:48:15.972 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31443]: starting. 2024-04-19T01:48:15.972 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31443]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-31442 2024-04-19T01:48:15.972 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31443]: created object 0... 2024-04-19T01:48:15.972 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31443]: created object 25... 2024-04-19T01:48:15.972 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31443]: created object 49... 2024-04-19T01:48:15.972 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31443]: finishing. 2024-04-19T01:48:15.972 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31443]: shutting down. 2024-04-19T01:48:17.576 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[31444]: starting. 2024-04-19T01:48:17.576 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[31444]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-31442 2024-04-19T01:48:17.577 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[31444]: shutting down. 2024-04-19T01:48:20.009 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:20.009 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31477]: starting. 2024-04-19T01:48:20.009 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31477]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-31442 2024-04-19T01:48:20.009 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31477]: created object 0... 2024-04-19T01:48:20.009 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31477]: created object 25... 2024-04-19T01:48:20.009 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31477]: created object 49... 2024-04-19T01:48:20.009 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31477]: finishing. 2024-04-19T01:48:20.010 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31477]: shutting down. 2024-04-19T01:48:20.273 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:20.274 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31479]: starting. 2024-04-19T01:48:20.274 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31479]: listing objects. 2024-04-19T01:48:20.274 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31479]: listed object 0... 2024-04-19T01:48:20.274 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31479]: listed object 25... 2024-04-19T01:48:20.274 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31479]: saw 50 objects 2024-04-19T01:48:20.274 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31479]: shutting down. 2024-04-19T01:48:20.610 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:20.611 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[31478]: starting. 2024-04-19T01:48:20.611 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[31478]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-31442 2024-04-19T01:48:20.611 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[31478]: shutting down. 2024-04-19T01:48:20.616 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:20.616 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:20.616 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:48:20.617 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:48:22.841 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31530]: starting. 2024-04-19T01:48:22.841 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31530]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-31529 2024-04-19T01:48:22.842 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31530]: created object 0... 2024-04-19T01:48:22.842 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31530]: created object 25... 2024-04-19T01:48:22.842 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31530]: created object 49... 2024-04-19T01:48:22.842 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31530]: finishing. 2024-04-19T01:48:22.842 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31530]: shutting down. 2024-04-19T01:48:23.634 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[31531]: starting. 2024-04-19T01:48:23.634 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[31531]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-31529 2024-04-19T01:48:23.634 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[31531]: shutting down. 2024-04-19T01:48:25.878 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:25.878 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31564]: starting. 2024-04-19T01:48:25.878 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31564]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-31529 2024-04-19T01:48:25.878 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31564]: created object 0... 2024-04-19T01:48:25.878 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31564]: created object 25... 2024-04-19T01:48:25.879 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31564]: created object 49... 2024-04-19T01:48:25.879 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31564]: finishing. 2024-04-19T01:48:25.879 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31564]: shutting down. 2024-04-19T01:48:25.941 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:25.941 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31566]: starting. 2024-04-19T01:48:25.941 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31566]: listing objects. 2024-04-19T01:48:25.941 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31566]: listed object 0... 2024-04-19T01:48:25.941 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31566]: listed object 25... 2024-04-19T01:48:25.941 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31566]: saw 50 objects 2024-04-19T01:48:25.941 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31566]: shutting down. 2024-04-19T01:48:26.686 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:26.686 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[31565]: starting. 2024-04-19T01:48:26.686 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[31565]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-31529 2024-04-19T01:48:26.686 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[31565]: shutting down. 2024-04-19T01:48:26.689 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:26.690 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:26.690 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:48:26.693 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:48:28.920 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31617]: starting. 2024-04-19T01:48:28.921 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31617]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-31616 2024-04-19T01:48:28.921 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31617]: created object 0... 2024-04-19T01:48:28.921 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31617]: created object 25... 2024-04-19T01:48:28.921 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31617]: created object 49... 2024-04-19T01:48:28.921 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31617]: finishing. 2024-04-19T01:48:28.921 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31617]: shutting down. 2024-04-19T01:48:29.720 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[31618]: starting. 2024-04-19T01:48:29.721 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[31618]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-31616 2024-04-19T01:48:29.721 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[31618]: shutting down. 2024-04-19T01:48:31.747 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:31.747 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31651]: starting. 2024-04-19T01:48:31.747 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31651]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-31616 2024-04-19T01:48:31.747 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31651]: created object 0... 2024-04-19T01:48:31.747 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31651]: created object 25... 2024-04-19T01:48:31.747 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31651]: created object 49... 2024-04-19T01:48:31.748 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31651]: finishing. 2024-04-19T01:48:31.748 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31651]: shutting down. 2024-04-19T01:48:31.805 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:31.805 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31653]: starting. 2024-04-19T01:48:31.805 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31653]: listing objects. 2024-04-19T01:48:31.805 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31653]: listed object 0... 2024-04-19T01:48:31.805 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31653]: listed object 25... 2024-04-19T01:48:31.805 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31653]: saw 50 objects 2024-04-19T01:48:31.805 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31653]: shutting down. 2024-04-19T01:48:32.540 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:32.540 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[31652]: starting. 2024-04-19T01:48:32.540 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[31652]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-31616 2024-04-19T01:48:32.540 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[31652]: shutting down. 2024-04-19T01:48:32.543 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:32.543 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:32.543 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:48:32.546 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:48:34.770 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31704]: starting. 2024-04-19T01:48:34.770 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31704]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-31703 2024-04-19T01:48:34.770 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31704]: created object 0... 2024-04-19T01:48:34.770 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31704]: created object 25... 2024-04-19T01:48:34.771 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31704]: created object 49... 2024-04-19T01:48:34.771 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31704]: finishing. 2024-04-19T01:48:34.771 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31704]: shutting down. 2024-04-19T01:48:35.571 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[31705]: starting. 2024-04-19T01:48:35.571 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[31705]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-31703 2024-04-19T01:48:35.571 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[31705]: shutting down. 2024-04-19T01:48:36.173 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-19T01:48:36.174 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-04-19T01:48:36.174 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-19T01:48:36.174 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-19T01:48:36.177 INFO:tasks.ceph.mon.c:Started 2024-04-19T01:48:36.177 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-19T01:48:36.177 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:48:36.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f434dfec640 10 monclient: get_monmap_and_config 2024-04-19T01:48:36.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f434dfec640 10 monclient: build_initial_monmap 2024-04-19T01:48:36.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f434dfec640 10 monclient: monmap: 2024-04-19T01:48:36.302 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:36.302 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:36.302 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:48:36.308157+0000 2024-04-19T01:48:36.302 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:48:36.308157+0000 2024-04-19T01:48:36.302 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:36.302 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:36.302 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:48:36.302 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:48:36.302 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:48:36.302 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:48:36.302 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:48:36.303 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:36.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f434dfec640 1 Processor -- start 2024-04-19T01:48:36.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f434dfec640 1 -- start start 2024-04-19T01:48:36.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f434dfec640 10 monclient: init 2024-04-19T01:48:36.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f434dfec640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:36.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f434dfec640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-04-19T01:48:36.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f434dfec640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4348137780 0x7f4348137b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:36.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f434dfec640 10 monclient(hunting): picked mon.noname-c con 0x7f4348137780 addr v2:172.21.15.71:3300/0 2024-04-19T01:48:36.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f434dfec640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f43481380a0 0x7f43480a45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:36.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f434dfec640 10 monclient(hunting): picked mon.noname-b con 0x7f43481380a0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:36.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f434dfec640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f43480a60b0 0x7f43480a4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:36.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f434dfec640 10 monclient(hunting): picked mon.noname-a con 0x7f43480a60b0 addr v2:172.21.15.62:3300/0 2024-04-19T01:48:36.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f434dfec640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:36.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f434dfec640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f434809bd20 con 0x7f43480a60b0 2024-04-19T01:48:36.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f434dfec640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:36.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f434dfec640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f434809c2e0 con 0x7f4348137780 2024-04-19T01:48:36.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f434dfec640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:36.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f434dfec640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f434809c000 con 0x7f43481380a0 2024-04-19T01:48:36.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f434dfec640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:36.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f434dfec640 10 monclient(hunting): authenticate will time out at 1302.127377s 2024-04-19T01:48:36.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f4346ffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f43481380a0 0x7f43480a45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:36.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f4347fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f43480a60b0 0x7f43480a4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:36.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f4347fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f43480a60b0 0x7f43480a4b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39254/0 (socket says 172.21.15.62:39254) 2024-04-19T01:48:36.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f4346ffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f43481380a0 0x7f43480a45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:50336/0 (socket says 172.21.15.62:50336) 2024-04-19T01:48:36.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f4347fff640 1 -- 172.21.15.62:0/3161844216 learned_addr learned my addr 172.21.15.62:0/3161844216 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:36.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f4347fff640 10 monclient(hunting): get_auth_request con 0x7f43480a60b0 auth_method 0 2024-04-19T01:48:36.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f4347fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:36.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f4347fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:36.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f4347fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:36.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f4346ffd640 10 monclient(hunting): get_auth_request con 0x7f43481380a0 auth_method 0 2024-04-19T01:48:36.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f4346ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:36.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f4346ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:36.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f4346ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:36.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f4347fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:36.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f4347fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:36.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f4346ffd640 1 -- 172.21.15.62:0/3161844216 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f43481380a0 msgr2=0x7f43480a45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:48:36.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f4346ffd640 1 -- 172.21.15.62:0/3161844216 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f43481380a0 msgr2=0x7f43480a45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:48:36.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f4347fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:36.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f4346ffd640 1 --2- 172.21.15.62:0/3161844216 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f43481380a0 0x7f43480a45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:48:36.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f4346ffd640 1 --2- 172.21.15.62:0/3161844216 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f43481380a0 0x7f43480a45d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:48:36.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f43477fe640 1 --2- 172.21.15.62:0/3161844216 >> v2:172.21.15.71:3300/0 conn(0x7f4348137780 0x7f4348137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:36.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f43477fe640 10 monclient(hunting): get_auth_request con 0x7f4348137780 auth_method 0 2024-04-19T01:48:36.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f43477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:36.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f43477fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:36.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f43477fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:36.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f4347fff640 10 monclient(hunting): handle_auth_done global_id 15198 payload 274 2024-04-19T01:48:36.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f4347fff640 1 -- 172.21.15.62:0/3161844216 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f43481380a0 msgr2=0x7f43480a45d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:48:36.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f4347fff640 1 --2- 172.21.15.62:0/3161844216 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f43481380a0 0x7f43480a45d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:36.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f4347fff640 1 -- 172.21.15.62:0/3161844216 >> v2:172.21.15.71:3300/0 conn(0x7f4348137780 msgr2=0x7f4348137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:36.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f4347fff640 1 --2- 172.21.15.62:0/3161844216 >> v2:172.21.15.71:3300/0 conn(0x7f4348137780 0x7f4348137b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:36.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f4347fff640 10 monclient: _finish_hunting 0 2024-04-19T01:48:36.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f4347fff640 1 monclient: found mon.noname-a 2024-04-19T01:48:36.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f4347fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:36.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f4347fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:48:36.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.302+0000 7f4347fff640 1 -- 172.21.15.62:0/3161844216 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4348073210 con 0x7f43480a60b0 2024-04-19T01:48:36.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f4347fff640 1 -- 172.21.15.62:0/3161844216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43480a60b0 msgr2=0x7f43480a4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:36.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f4347fff640 1 --2- 172.21.15.62:0/3161844216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43480a60b0 0x7f43480a4b10 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f43480a5a70 tx=0x7f434001c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c89bc12130dd14cd server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:36.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f43467fc640 1 -- 172.21.15.62:0/3161844216 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43400a5640 con 0x7f43480a60b0 2024-04-19T01:48:36.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f43467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:36.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f43467fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:48:36.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f43467fc640 10 monclient: dump: 2024-04-19T01:48:36.315 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:36.315 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:36.315 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:36.315 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:36.315 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:36.315 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:36.315 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:36.315 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:36.315 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:36.315 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:36.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f43467fc640 10 monclient: _finish_auth 0 2024-04-19T01:48:36.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f43467fc640 10 monclient: _check_auth_tickets 2024-04-19T01:48:36.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f43467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:36.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 5 monclient: authenticate success, global_id 15198 2024-04-19T01:48:36.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:48:36.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f43467fc640 1 -- 172.21.15.62:0/3161844216 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43400a57e0 con 0x7f43480a60b0 2024-04-19T01:48:36.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f43467fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:36.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f43467fc640 1 -- 172.21.15.62:0/3161844216 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43400a5bb0 con 0x7f43480a60b0 2024-04-19T01:48:36.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 10 monclient: get_monmap_and_config success 2024-04-19T01:48:36.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 4 set_mon_vals no callback set 2024-04-19T01:48:36.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f43467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:36.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f43467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:48:36.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f43467fc640 10 monclient: dump: 2024-04-19T01:48:36.316 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:36.316 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:36.316 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:36.316 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:36.316 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:36.316 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:36.317 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:36.317 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:36.317 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:36.317 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:36.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 10 monclient: shutdown 2024-04-19T01:48:36.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:36.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 1 -- 172.21.15.62:0/3161844216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43480a60b0 msgr2=0x7f43480a4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:36.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 1 --2- 172.21.15.62:0/3161844216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43480a60b0 0x7f43480a4b10 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f43480a5a70 tx=0x7f434001c8b0 comp rx=0 tx=0).stop 2024-04-19T01:48:36.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 1 -- 172.21.15.62:0/3161844216 shutdown_connections 2024-04-19T01:48:36.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 1 --2- 172.21.15.62:0/3161844216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43480a60b0 0x7f43480a4b10 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:36.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 1 --2- 172.21.15.62:0/3161844216 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f43481380a0 0x7f43480a45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:36.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 1 --2- 172.21.15.62:0/3161844216 >> v2:172.21.15.71:3300/0 conn(0x7f4348137780 0x7f4348137b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:36.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 1 -- 172.21.15.62:0/3161844216 >> 172.21.15.62:0/3161844216 conn(0x7f43480aa1d0 msgr2=0x7f43480a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:36.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 1 -- 172.21.15.62:0/3161844216 shutdown_connections 2024-04-19T01:48:36.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 1 -- 172.21.15.62:0/3161844216 wait complete. 2024-04-19T01:48:36.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 10 monclient: build_initial_monmap 2024-04-19T01:48:36.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 10 monclient: monmap: 2024-04-19T01:48:36.318 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:36.318 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:48:36.318 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:48:36.318 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:48:36.318 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:36.318 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:36.318 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:48:36.318 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:48:36.318 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:48:36.318 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:36.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 1 Processor -- start 2024-04-19T01:48:36.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 1 -- start start 2024-04-19T01:48:36.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 10 monclient: init 2024-04-19T01:48:36.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:36.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-19T01:48:36.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43480a60b0 0x7f43480a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:36.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 10 monclient(hunting): picked mon.noname-a con 0x7f43480a60b0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:48:36.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4348137780 0x7f43480a1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:36.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 10 monclient(hunting): picked mon.noname-c con 0x7f4348137780 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:36.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43481380a0 0x7f4348136590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:36.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 10 monclient(hunting): picked mon.noname-b con 0x7f43481380a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:48:36.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:36.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4348141160 con 0x7f43480a60b0 2024-04-19T01:48:36.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:36.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f43480a5460 con 0x7f43481380a0 2024-04-19T01:48:36.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:36.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f43481412e0 con 0x7f4348137780 2024-04-19T01:48:36.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:36.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f434dfec640 10 monclient(hunting): authenticate will time out at 1302.134412s 2024-04-19T01:48:36.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f4346ffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4348137780 0x7f43480a1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:36.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f4346ffd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4348137780 0x7f43480a1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:50352/0 (socket says 172.21.15.62:50352) 2024-04-19T01:48:36.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f4346ffd640 1 -- 172.21.15.62:0/4247636020 learned_addr learned my addr 172.21.15.62:0/4247636020 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:36.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f4346ffd640 10 monclient(hunting): get_auth_request con 0x7f4348137780 auth_method 0 2024-04-19T01:48:36.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f4346ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:36.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f4346ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:36.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f4346ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:36.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f43477fe640 1 --2- 172.21.15.62:0/4247636020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43480a60b0 0x7f43480a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:36.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f43477fe640 10 monclient(hunting): get_auth_request con 0x7f43480a60b0 auth_method 0 2024-04-19T01:48:36.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f43477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:36.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f43477fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:36.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f43477fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:36.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f4346ffd640 1 -- 172.21.15.62:0/4247636020 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4348137780 msgr2=0x7f43480a1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-19T01:48:36.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f4346ffd640 1 -- 172.21.15.62:0/4247636020 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4348137780 msgr2=0x7f43480a1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:48:36.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f4346ffd640 1 --2- 172.21.15.62:0/4247636020 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4348137780 0x7f43480a1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:48:36.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f43477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:36.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f43477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:36.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f4346ffd640 1 --2- 172.21.15.62:0/4247636020 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4348137780 0x7f43480a1bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:48:36.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f43477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:36.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f4347fff640 1 --2- 172.21.15.62:0/4247636020 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43481380a0 0x7f4348136590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:36.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f4347fff640 10 monclient(hunting): get_auth_request con 0x7f43481380a0 auth_method 0 2024-04-19T01:48:36.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f4347fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:36.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f4347fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:36.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f4347fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:36.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f43477fe640 10 monclient(hunting): handle_auth_done global_id 15201 payload 824 2024-04-19T01:48:36.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f43477fe640 1 -- 172.21.15.62:0/4247636020 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4348137780 msgr2=0x7f43480a1bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:48:36.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f43477fe640 1 --2- 172.21.15.62:0/4247636020 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4348137780 0x7f43480a1bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:36.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f43477fe640 1 -- 172.21.15.62:0/4247636020 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43481380a0 msgr2=0x7f4348136590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:36.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f43477fe640 1 --2- 172.21.15.62:0/4247636020 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43481380a0 0x7f4348136590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:36.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f43477fe640 10 monclient: _finish_hunting 0 2024-04-19T01:48:36.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f43477fe640 1 monclient: found mon.noname-a 2024-04-19T01:48:36.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f43477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:36.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f43477fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:48:36.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f43477fe640 1 -- 172.21.15.62:0/4247636020 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f434813de70 con 0x7f43480a60b0 2024-04-19T01:48:36.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.310+0000 7f4347fff640 1 --2- 172.21.15.62:0/4247636020 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43481380a0 0x7f4348136590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T01:48:36.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.314+0000 7f43477fe640 1 -- 172.21.15.62:0/4247636020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43480a60b0 msgr2=0x7f43480a0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:36.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.314+0000 7f43477fe640 1 --2- 172.21.15.62:0/4247636020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43480a60b0 0x7f43480a0fd0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f433000d5f0 tx=0x7f433000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:36.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.314+0000 7f433ffff640 1 -- 172.21.15.62:0/4247636020 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4330012800 con 0x7f43480a60b0 2024-04-19T01:48:36.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.314+0000 7f433ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:36.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.314+0000 7f433ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:48:36.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.314+0000 7f433ffff640 10 monclient: dump: 2024-04-19T01:48:36.322 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:36.323 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:36.323 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:36.323 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:36.323 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:36.323 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:36.323 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:36.323 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:36.323 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:36.323 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:36.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.314+0000 7f433ffff640 10 monclient: _finish_auth 0 2024-04-19T01:48:36.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.314+0000 7f433ffff640 10 monclient: _check_auth_tickets 2024-04-19T01:48:36.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.314+0000 7f433ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:36.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.314+0000 7f433ffff640 1 -- 172.21.15.62:0/4247636020 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4330019070 con 0x7f43480a60b0 2024-04-19T01:48:36.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.314+0000 7f433ffff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:36.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.314+0000 7f433ffff640 1 -- 172.21.15.62:0/4247636020 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4330012bc0 con 0x7f43480a60b0 2024-04-19T01:48:36.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.314+0000 7f433ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:36.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.314+0000 7f433ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:48:36.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.314+0000 7f433ffff640 10 monclient: dump: 2024-04-19T01:48:36.324 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:36.324 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:36.324 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:36.324 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:36.324 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:36.324 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:36.324 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:36.325 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:36.325 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:36.325 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:36.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.314+0000 7f43457fa640 4 set_mon_vals no callback set 2024-04-19T01:48:36.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.314+0000 7f434dfec640 5 monclient: authenticate success, global_id 15201 2024-04-19T01:48:36.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.314+0000 7f434dfec640 10 monclient: _renew_subs 2024-04-19T01:48:36.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.314+0000 7f434dfec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:36.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.314+0000 7f434dfec640 1 -- 172.21.15.62:0/4247636020 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43480a2710 con 0x7f43480a60b0 2024-04-19T01:48:36.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.314+0000 7f434dfec640 10 monclient: _renew_subs 2024-04-19T01:48:36.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.314+0000 7f434dfec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:36.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.314+0000 7f434dfec640 1 -- 172.21.15.62:0/4247636020 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4348120a00 con 0x7f43480a60b0 2024-04-19T01:48:36.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.314+0000 7f434dfec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:36.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.314+0000 7f434dfec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:36.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.314+0000 7f434dfec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:36.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.314+0000 7f434dfec640 1 -- 172.21.15.62:0/4247636020 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4348068ab0 con 0x7f43480a60b0 2024-04-19T01:48:36.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.314+0000 7f433ffff640 1 -- 172.21.15.62:0/4247636020 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f433001e460 con 0x7f43480a60b0 2024-04-19T01:48:36.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.314+0000 7f433ffff640 1 --2- 172.21.15.62:0/4247636020 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4328036420 0x7f43280388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:36.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.314+0000 7f433ffff640 1 -- 172.21.15.62:0/4247636020 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(315..315 src has 115..315) ==== 4346+0+0 (secure 0 0 0) 0x7f4330049870 con 0x7f43480a60b0 2024-04-19T01:48:36.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.314+0000 7f4346ffd640 1 --2- 172.21.15.62:0/4247636020 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4328036420 0x7f43280388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:36.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.314+0000 7f4346ffd640 10 monclient: get_auth_request con 0x7f4328036420 auth_method 0 2024-04-19T01:48:36.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.318+0000 7f4346ffd640 1 --2- 172.21.15.62:0/4247636020 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4328036420 0x7f43280388e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f43480a3b50 tx=0x7f4338009440 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:36.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.322+0000 7f433ffff640 1 -- 172.21.15.62:0/4247636020 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f433001d7c0 con 0x7f43480a60b0 2024-04-19T01:48:36.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.322+0000 7f433ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:36.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.322+0000 7f433ffff640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:48:36.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.550+0000 7f434dfec640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:48:36.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.550+0000 7f434dfec640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:48:36.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.550+0000 7f434dfec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:36.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.550+0000 7f434dfec640 1 -- 172.21.15.62:0/4247636020 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4348068710 con 0x7f43480a60b0 2024-04-19T01:48:36.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.550+0000 7f433ffff640 1 -- 172.21.15.62:0/4247636020 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f4330012d60 con 0x7f43480a60b0 2024-04-19T01:48:36.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.550+0000 7f433ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:48:36.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.550+0000 7f433ffff640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:48:36.550 INFO:teuthology.orchestra.run.smithi062.stdout:{"election_epoch":38,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":91,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","modified":"2024-04-19T01:41:38.497795Z","created":"2024-04-19T01:41:38.497795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:48:36.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.554+0000 7f434dfec640 1 -- 172.21.15.62:0/4247636020 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4328036420 msgr2=0x7f43280388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:36.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.554+0000 7f434dfec640 1 --2- 172.21.15.62:0/4247636020 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4328036420 0x7f43280388e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f43480a3b50 tx=0x7f4338009440 comp rx=0 tx=0).stop 2024-04-19T01:48:36.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.554+0000 7f434dfec640 10 monclient: shutdown 2024-04-19T01:48:36.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.554+0000 7f434dfec640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:36.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.554+0000 7f434dfec640 1 -- 172.21.15.62:0/4247636020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43480a60b0 msgr2=0x7f43480a0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:36.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.554+0000 7f434dfec640 1 --2- 172.21.15.62:0/4247636020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43480a60b0 0x7f43480a0fd0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f433000d5f0 tx=0x7f433000dab0 comp rx=0 tx=0).stop 2024-04-19T01:48:36.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.554+0000 7f434dfec640 1 -- 172.21.15.62:0/4247636020 shutdown_connections 2024-04-19T01:48:36.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.554+0000 7f434dfec640 1 --2- 172.21.15.62:0/4247636020 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4328036420 0x7f43280388e0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:36.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.554+0000 7f434dfec640 1 --2- 172.21.15.62:0/4247636020 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4348137780 0x7f43480a1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:36.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.554+0000 7f434dfec640 1 --2- 172.21.15.62:0/4247636020 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43481380a0 0x7f4348136590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:36.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.554+0000 7f434dfec640 1 --2- 172.21.15.62:0/4247636020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43480a60b0 0x7f43480a0fd0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:36.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.554+0000 7f434dfec640 1 -- 172.21.15.62:0/4247636020 >> 172.21.15.62:0/4247636020 conn(0x7f43480aa1d0 msgr2=0x7f43480a8260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:36.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.554+0000 7f434dfec640 1 -- 172.21.15.62:0/4247636020 shutdown_connections 2024-04-19T01:48:36.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:36.554+0000 7f434dfec640 1 -- 172.21.15.62:0/4247636020 wait complete. 2024-04-19T01:48:37.800 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:37.800 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31738]: starting. 2024-04-19T01:48:37.800 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31738]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-31703 2024-04-19T01:48:37.800 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31738]: created object 0... 2024-04-19T01:48:37.800 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31738]: created object 25... 2024-04-19T01:48:37.800 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31738]: created object 49... 2024-04-19T01:48:37.800 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31738]: finishing. 2024-04-19T01:48:37.800 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31738]: shutting down. 2024-04-19T01:48:38.065 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:38.065 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31740]: starting. 2024-04-19T01:48:38.066 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31740]: listing objects. 2024-04-19T01:48:38.066 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31740]: listed object 0... 2024-04-19T01:48:38.066 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31740]: listed object 25... 2024-04-19T01:48:38.066 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31740]: saw 50 objects 2024-04-19T01:48:38.066 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31740]: shutting down. 2024-04-19T01:48:38.597 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:38.597 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[31739]: starting. 2024-04-19T01:48:38.597 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[31739]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-31703 2024-04-19T01:48:38.597 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[31739]: shutting down. 2024-04-19T01:48:38.599 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:38.599 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:38.599 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:48:38.602 DEBUG:teuthology.orchestra.run.smithi071:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-19T01:48:39.571 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:48:39.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.710+0000 7f0654709640 10 monclient: get_monmap_and_config 2024-04-19T01:48:39.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.710+0000 7f0654709640 10 monclient: build_initial_monmap 2024-04-19T01:48:39.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.710+0000 7f0654709640 10 monclient: monmap: 2024-04-19T01:48:39.710 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:39.711 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:39.711 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:48:39.715378+0000 2024-04-19T01:48:39.711 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:48:39.715378+0000 2024-04-19T01:48:39.711 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:39.711 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:39.711 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:48:39.711 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:48:39.711 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:48:39.712 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:48:39.712 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:48:39.712 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:39.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.710+0000 7f0654709640 1 Processor -- start 2024-04-19T01:48:39.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.710+0000 7f0654709640 1 -- start start 2024-04-19T01:48:39.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.710+0000 7f0654709640 10 monclient: init 2024-04-19T01:48:39.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.710+0000 7f0654709640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:39.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.710+0000 7f0654709640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-04-19T01:48:39.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.710+0000 7f0654709640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f064c0a2460 0x7f064c0a2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:39.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.710+0000 7f0654709640 10 monclient(hunting): picked mon.noname-b con 0x7f064c0a2460 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:39.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.710+0000 7f0654709640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f064c075fe0 addr v1:172.21.15.71:6789/0 2024-04-19T01:48:39.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.710+0000 7f0654709640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f064c0a2d80 addr v1:172.21.15.62:6789/0 2024-04-19T01:48:39.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.710+0000 7f0654709640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f064c135030 con 0x7f064c0a2d80 2024-04-19T01:48:39.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.710+0000 7f0654709640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f064c09ff10 con 0x7f064c075fe0 2024-04-19T01:48:39.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.710+0000 7f0654709640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:39.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.710+0000 7f0654709640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f064c0a0840 con 0x7f064c0a2460 2024-04-19T01:48:39.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.710+0000 7f0654709640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:39.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.710+0000 7f0654709640 10 monclient(hunting): authenticate will time out at 1305.534943s 2024-04-19T01:48:39.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.710+0000 7f06524a5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f064c0a2460 0x7f064c0a2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:39.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.710+0000 7f06524a5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f064c0a2460 0x7f064c0a2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:50358/0 (socket says 172.21.15.62:50358) 2024-04-19T01:48:39.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.710+0000 7f06524a5640 1 -- 172.21.15.62:0/2780778964 learned_addr learned my addr 172.21.15.62:0/2780778964 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:39.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.710+0000 7f06524a5640 10 monclient(hunting): get_auth_request con 0x7f064c0a2460 auth_method 0 2024-04-19T01:48:39.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.710+0000 7f06524a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:39.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.710+0000 7f06524a5640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:39.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.710+0000 7f06524a5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:39.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.710+0000 7f0652ca6640 1 --1- 172.21.15.62:0/2780778964 >> v1:172.21.15.62:6789/0 conn(0x7f064c0a2d80 0x7f064c134920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:39408/0 (socket says 172.21.15.62:39408) 2024-04-19T01:48:39.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.710+0000 7f06524a5640 1 -- 172.21.15.62:0/2780778964 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f064c0a2460 msgr2=0x7f064c0a2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:48:39.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.710+0000 7f06524a5640 1 -- 172.21.15.62:0/2780778964 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f064c0a2460 msgr2=0x7f064c0a2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:48:39.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.710+0000 7f06524a5640 1 --2- 172.21.15.62:0/2780778964 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f064c0a2460 0x7f064c0a2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:48:39.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.710+0000 7f06524a5640 1 --2- 172.21.15.62:0/2780778964 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f064c0a2460 0x7f064c0a2840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:48:39.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0652ca6640 1 -- 172.21.15.62:0/2780778964 >> v1:172.21.15.62:6789/0 conn(0x7f064c0a2d80 legacy=0x7f064c134920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:39.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 1 -- 172.21.15.62:0/2780778964 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2372215418 0 0) 0x7f064c135030 con 0x7f064c0a2d80 2024-04-19T01:48:39.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 10 monclient(hunting): _negotiate 2024-04-19T01:48:39.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:39.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:39.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 10 monclient(hunting): my global_id is 15216 2024-04-19T01:48:39.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0651ca4640 1 -- 172.21.15.62:0/2780778964 >> v1:172.21.15.71:6789/0 conn(0x7f064c075fe0 legacy=0x7f064c134210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:39.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 1 -- 172.21.15.62:0/2780778964 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0638003530 con 0x7f064c0a2d80 2024-04-19T01:48:39.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 1 -- 172.21.15.62:0/2780778964 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1864824915 0 0) 0x7f0638003530 con 0x7f064c0a2d80 2024-04-19T01:48:39.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 1 -- 172.21.15.62:0/2780778964 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f064c0a2460 msgr2=0x7f064c0a2840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:48:39.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 1 --2- 172.21.15.62:0/2780778964 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f064c0a2460 0x7f064c0a2840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:39.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 1 -- 172.21.15.62:0/2780778964 >> v1:172.21.15.71:6789/0 conn(0x7f064c075fe0 legacy=0x7f064c134210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:39.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 10 monclient: _finish_hunting 0 2024-04-19T01:48:39.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 1 monclient: found mon.noname-a-legacy 2024-04-19T01:48:39.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:39.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.62:6789/0 2024-04-19T01:48:39.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 1 -- 172.21.15.62:0/2780778964 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f064c0a0b20 con 0x7f064c0a2d80 2024-04-19T01:48:39.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 10 monclient: _finish_auth 0 2024-04-19T01:48:39.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 10 monclient: _check_auth_tickets 2024-04-19T01:48:39.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:39.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0654709640 5 monclient: authenticate success, global_id 15216 2024-04-19T01:48:39.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0654709640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:48:39.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 1 -- 172.21.15.62:0/2780778964 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1244427781 0 0) 0x7f06480019b0 con 0x7f064c0a2d80 2024-04-19T01:48:39.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:39.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-19T01:48:39.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 10 monclient: dump: 2024-04-19T01:48:39.731 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:39.732 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:39.732 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:39.732 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:39.732 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:39.732 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:39.732 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:39.732 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:39.732 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:39.732 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:39.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] but i'm connected to v1:172.21.15.62:6789/0, reconnecting 2024-04-19T01:48:39.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:39.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 1 -- 172.21.15.62:0/2780778964 >> v1:172.21.15.62:6789/0 conn(0x7f064c0a2d80 legacy=0x7f064c134920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:39.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-19T01:48:39.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 1 --2- 172.21.15.62:0/2780778964 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0638003ac0 0x7f0638003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:39.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 10 monclient(hunting): picked mon.b con 0x7f0638003ac0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:48:39.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 1 --2- 172.21.15.62:0/2780778964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0638004810 0x7f0638004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:39.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 10 monclient(hunting): picked mon.a con 0x7f0638004810 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:48:39.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 1 --2- 172.21.15.62:0/2780778964 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f064c0a2460 0x7f063800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:39.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 10 monclient(hunting): picked mon.c con 0x7f064c0a2460 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:39.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:39.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 1 -- 172.21.15.62:0/2780778964 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f064c0a0840 con 0x7f0638004810 2024-04-19T01:48:39.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:39.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 1 -- 172.21.15.62:0/2780778964 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f0638005630 con 0x7f0638003ac0 2024-04-19T01:48:39.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:39.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 1 -- 172.21.15.62:0/2780778964 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f063800bf40 con 0x7f064c0a2460 2024-04-19T01:48:39.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:39.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06514a3640 10 monclient(hunting): _finish_auth 0 2024-04-19T01:48:39.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0654709640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-19T01:48:39.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0651ca4640 1 --2- 172.21.15.62:0/2780778964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0638004810 0x7f0638004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:39.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0651ca4640 10 monclient(hunting): get_auth_request con 0x7f0638004810 auth_method 0 2024-04-19T01:48:39.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0651ca4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:39.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0651ca4640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:39.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0652ca6640 1 --2- 172.21.15.62:0/2780778964 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f064c0a2460 0x7f063800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:39.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0651ca4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:48:39.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0652ca6640 10 monclient(hunting): get_auth_request con 0x7f064c0a2460 auth_method 0 2024-04-19T01:48:39.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0652ca6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:39.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0652ca6640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:39.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0652ca6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:48:39.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06524a5640 1 --2- 172.21.15.62:0/2780778964 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0638003ac0 0x7f0638003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:39.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0651ca4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:39.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0651ca4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:39.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0651ca4640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T01:48:39.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06524a5640 10 monclient(hunting): get_auth_request con 0x7f0638003ac0 auth_method 0 2024-04-19T01:48:39.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06524a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:39.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06524a5640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:39.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06524a5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:48:39.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0652ca6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:39.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0652ca6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:39.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0652ca6640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T01:48:39.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06524a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:39.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06524a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:39.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f06524a5640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T01:48:39.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0651ca4640 10 monclient(hunting): handle_auth_done global_id 15216 payload 293 2024-04-19T01:48:39.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0651ca4640 1 -- 172.21.15.62:0/2780778964 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f064c0a2460 msgr2=0x7f063800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:39.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0651ca4640 1 --2- 172.21.15.62:0/2780778964 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f064c0a2460 0x7f063800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:39.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0651ca4640 1 -- 172.21.15.62:0/2780778964 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0638003ac0 msgr2=0x7f0638003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:39.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0651ca4640 1 --2- 172.21.15.62:0/2780778964 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0638003ac0 0x7f0638003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:39.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0651ca4640 10 monclient: _finish_hunting 0 2024-04-19T01:48:39.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0651ca4640 1 monclient: found mon.a 2024-04-19T01:48:39.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0651ca4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:39.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0651ca4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:39.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0651ca4640 1 -- 172.21.15.62:0/2780778964 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0638003530 con 0x7f0638004810 2024-04-19T01:48:39.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0651ca4640 10 monclient: _finish_auth 0 2024-04-19T01:48:39.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0651ca4640 10 monclient: _check_auth_tickets 2024-04-19T01:48:39.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.714+0000 7f0651ca4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:39.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.718+0000 7f0651ca4640 1 -- 172.21.15.62:0/2780778964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0638004810 msgr2=0x7f0638004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:39.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.718+0000 7f0651ca4640 1 --2- 172.21.15.62:0/2780778964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0638004810 0x7f0638004c90 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f064c0763e0 tx=0x7f064001c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4637ed6aeadc770 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:39.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.718+0000 7f06514a3640 1 -- 172.21.15.62:0/2780778964 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0640001e70 con 0x7f0638004810 2024-04-19T01:48:39.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.718+0000 7f06514a3640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:39.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.718+0000 7f06514a3640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:48:39.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.718+0000 7f06514a3640 10 monclient: dump: 2024-04-19T01:48:39.741 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:39.742 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:39.742 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:39.742 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:39.742 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:39.742 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:39.742 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:39.742 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:39.743 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:39.743 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:39.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.718+0000 7f0654709640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:48:39.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.718+0000 7f06514a3640 1 -- 172.21.15.62:0/2780778964 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06400a9070 con 0x7f0638004810 2024-04-19T01:48:39.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.718+0000 7f06514a3640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:39.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.718+0000 7f0654709640 10 monclient: get_monmap_and_config success 2024-04-19T01:48:39.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.718+0000 7f06514a3640 1 -- 172.21.15.62:0/2780778964 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06400a55d0 con 0x7f0638004810 2024-04-19T01:48:39.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.718+0000 7f0654709640 4 set_mon_vals no callback set 2024-04-19T01:48:39.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.718+0000 7f06514a3640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:39.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.718+0000 7f06514a3640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:48:39.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.718+0000 7f06514a3640 10 monclient: dump: 2024-04-19T01:48:39.747 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:39.747 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:39.747 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:39.747 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:39.747 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:39.748 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:39.748 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:39.748 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:39.748 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:39.748 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:39.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.718+0000 7f0654709640 10 monclient: shutdown 2024-04-19T01:48:39.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.718+0000 7f0654709640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:39.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.718+0000 7f0654709640 1 -- 172.21.15.62:0/2780778964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0638004810 msgr2=0x7f0638004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:39.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.718+0000 7f0654709640 1 --2- 172.21.15.62:0/2780778964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0638004810 0x7f0638004c90 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f064c0763e0 tx=0x7f064001c990 comp rx=0 tx=0).stop 2024-04-19T01:48:39.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.718+0000 7f0652ca6640 1 -- 172.21.15.62:0/2780778964 reap_dead start 2024-04-19T01:48:39.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.718+0000 7f0654709640 1 -- 172.21.15.62:0/2780778964 shutdown_connections 2024-04-19T01:48:39.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.718+0000 7f0654709640 1 -- 172.21.15.62:0/2780778964 >> 172.21.15.62:0/2780778964 conn(0x7f064c0a7050 msgr2=0x7f064c0a3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:39.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.718+0000 7f0654709640 1 -- 172.21.15.62:0/2780778964 shutdown_connections 2024-04-19T01:48:39.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.718+0000 7f0654709640 1 -- 172.21.15.62:0/2780778964 wait complete. 2024-04-19T01:48:39.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0654709640 10 monclient: build_initial_monmap 2024-04-19T01:48:39.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0654709640 10 monclient: monmap: 2024-04-19T01:48:39.749 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:39.749 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:48:39.749 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:48:39.749 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:48:39.749 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:39.749 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:39.749 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:48:39.749 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:48:39.749 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:48:39.750 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:39.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0654709640 1 Processor -- start 2024-04-19T01:48:39.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0654709640 1 -- start start 2024-04-19T01:48:39.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0654709640 10 monclient: init 2024-04-19T01:48:39.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0654709640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:39.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0654709640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-19T01:48:39.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0654709640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0638004810 0x7f064c1cf2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:39.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0654709640 10 monclient(hunting): picked mon.noname-b con 0x7f0638004810 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:48:39.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0654709640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f064c133970 0x7f064c13dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:39.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0654709640 10 monclient(hunting): picked mon.noname-c con 0x7f064c133970 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:39.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0654709640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f064c133df0 0x7f064c13e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:39.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0654709640 10 monclient(hunting): picked mon.noname-a con 0x7f064c133df0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:48:39.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0654709640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:39.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0654709640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f064c0a4040 con 0x7f064c133df0 2024-04-19T01:48:39.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0654709640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:39.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0654709640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f064c0a3ec0 con 0x7f0638004810 2024-04-19T01:48:39.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0654709640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:39.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0654709640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f064c141190 con 0x7f064c133970 2024-04-19T01:48:39.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0654709640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:39.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0654709640 10 monclient(hunting): authenticate will time out at 1305.545249s 2024-04-19T01:48:39.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0651ca4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f064c133970 0x7f064c13dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:39.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0652ca6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f064c133df0 0x7f064c13e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:39.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0651ca4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f064c133970 0x7f064c13dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:50382/0 (socket says 172.21.15.62:50382) 2024-04-19T01:48:39.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0651ca4640 1 -- 172.21.15.62:0/1625974650 learned_addr learned my addr 172.21.15.62:0/1625974650 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:39.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0651ca4640 10 monclient(hunting): get_auth_request con 0x7f064c133970 auth_method 0 2024-04-19T01:48:39.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0652ca6640 1 --2- 172.21.15.62:0/1625974650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f064c133df0 0x7f064c13e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39328/0 (socket says 172.21.15.62:39328) 2024-04-19T01:48:39.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0651ca4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:39.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0651ca4640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:39.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0651ca4640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:39.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0652ca6640 10 monclient(hunting): get_auth_request con 0x7f064c133df0 auth_method 0 2024-04-19T01:48:39.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0652ca6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:39.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0652ca6640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:39.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0652ca6640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:39.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0652ca6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:39.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0651ca4640 1 -- 172.21.15.62:0/1625974650 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f064c133970 msgr2=0x7f064c13dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-19T01:48:39.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0652ca6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:39.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0651ca4640 1 -- 172.21.15.62:0/1625974650 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f064c133970 msgr2=0x7f064c13dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:48:39.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0651ca4640 1 --2- 172.21.15.62:0/1625974650 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f064c133970 0x7f064c13dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:48:39.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0652ca6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:39.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0651ca4640 1 --2- 172.21.15.62:0/1625974650 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f064c133970 0x7f064c13dd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:48:39.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f06524a5640 1 --2- 172.21.15.62:0/1625974650 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0638004810 0x7f064c1cf2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:39.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f06524a5640 10 monclient(hunting): get_auth_request con 0x7f0638004810 auth_method 0 2024-04-19T01:48:39.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f06524a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:39.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f06524a5640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:39.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f06524a5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:39.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0652ca6640 10 monclient(hunting): handle_auth_done global_id 15219 payload 824 2024-04-19T01:48:39.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0652ca6640 1 -- 172.21.15.62:0/1625974650 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f064c133970 msgr2=0x7f064c13dd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:48:39.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0652ca6640 1 --2- 172.21.15.62:0/1625974650 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f064c133970 0x7f064c13dd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:39.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0652ca6640 1 -- 172.21.15.62:0/1625974650 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0638004810 msgr2=0x7f064c1cf2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:39.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0652ca6640 1 --2- 172.21.15.62:0/1625974650 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0638004810 0x7f064c1cf2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:39.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0652ca6640 10 monclient: _finish_hunting 0 2024-04-19T01:48:39.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0652ca6640 1 monclient: found mon.noname-a 2024-04-19T01:48:39.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0652ca6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:39.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0652ca6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:48:39.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f0652ca6640 1 -- 172.21.15.62:0/1625974650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f064c0a5aa0 con 0x7f064c133df0 2024-04-19T01:48:39.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.722+0000 7f06524a5640 1 --2- 172.21.15.62:0/1625974650 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0638004810 0x7f064c1cf2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T01:48:39.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.726+0000 7f0652ca6640 1 -- 172.21.15.62:0/1625974650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f064c133df0 msgr2=0x7f064c13e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:39.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.726+0000 7f0652ca6640 1 --2- 172.21.15.62:0/1625974650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f064c133df0 0x7f064c13e290 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f064c0754e0 tx=0x7f0648004bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:39.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.726+0000 7f063effd640 1 -- 172.21.15.62:0/1625974650 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064800f8b0 con 0x7f064c133df0 2024-04-19T01:48:39.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.726+0000 7f063effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:39.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.726+0000 7f063effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:48:39.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.726+0000 7f063effd640 10 monclient: dump: 2024-04-19T01:48:39.761 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:39.761 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:39.761 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:39.761 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:39.761 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:39.761 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:39.761 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:39.761 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:39.762 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:39.762 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:39.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.726+0000 7f063effd640 10 monclient: _finish_auth 0 2024-04-19T01:48:39.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.726+0000 7f063effd640 10 monclient: _check_auth_tickets 2024-04-19T01:48:39.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.726+0000 7f063effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:39.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.726+0000 7f063effd640 1 -- 172.21.15.62:0/1625974650 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0648018070 con 0x7f064c133df0 2024-04-19T01:48:39.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.726+0000 7f0654709640 5 monclient: authenticate success, global_id 15219 2024-04-19T01:48:39.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.726+0000 7f0654709640 10 monclient: _renew_subs 2024-04-19T01:48:39.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.726+0000 7f0654709640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:39.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.726+0000 7f0654709640 1 -- 172.21.15.62:0/1625974650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f064c071ab0 con 0x7f064c133df0 2024-04-19T01:48:39.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.726+0000 7f063effd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:39.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.726+0000 7f063effd640 1 -- 172.21.15.62:0/1625974650 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064800fc70 con 0x7f064c133df0 2024-04-19T01:48:39.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.726+0000 7f063effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:39.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.726+0000 7f063effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:48:39.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.726+0000 7f063effd640 10 monclient: dump: 2024-04-19T01:48:39.763 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:39.763 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:39.763 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:39.763 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:39.763 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:39.763 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:39.763 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:39.763 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:39.763 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:39.763 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:39.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.726+0000 7f0654709640 10 monclient: _renew_subs 2024-04-19T01:48:39.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.726+0000 7f0654709640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:39.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.726+0000 7f0654709640 1 -- 172.21.15.62:0/1625974650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f064c11ea70 con 0x7f064c133df0 2024-04-19T01:48:39.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.726+0000 7f0650ca2640 4 set_mon_vals no callback set 2024-04-19T01:48:39.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.726+0000 7f0654709640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:39.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.726+0000 7f0654709640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:39.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.726+0000 7f0654709640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:39.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.726+0000 7f0654709640 1 -- 172.21.15.62:0/1625974650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f064c068ab0 con 0x7f064c133df0 2024-04-19T01:48:39.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.726+0000 7f063effd640 1 -- 172.21.15.62:0/1625974650 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f064801d460 con 0x7f064c133df0 2024-04-19T01:48:39.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.726+0000 7f063effd640 1 --2- 172.21.15.62:0/1625974650 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0630036470 0x7f0630038930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:39.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.726+0000 7f063effd640 1 -- 172.21.15.62:0/1625974650 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(319..319 src has 115..319) ==== 4738+0+0 (secure 0 0 0) 0x7f06480492a0 con 0x7f064c133df0 2024-04-19T01:48:39.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.726+0000 7f06524a5640 1 --2- 172.21.15.62:0/1625974650 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0630036470 0x7f0630038930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:39.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.726+0000 7f06524a5640 10 monclient: get_auth_request con 0x7f0630036470 auth_method 0 2024-04-19T01:48:39.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.730+0000 7f06524a5640 1 --2- 172.21.15.62:0/1625974650 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0630036470 0x7f0630038930 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0644009100 tx=0x7f0644008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:39.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.730+0000 7f063effd640 1 -- 172.21.15.62:0/1625974650 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f064800fe10 con 0x7f064c133df0 2024-04-19T01:48:39.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.730+0000 7f063effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:39.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.730+0000 7f063effd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:48:39.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.954+0000 7f0654709640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:48:39.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.954+0000 7f0654709640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:48:39.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.954+0000 7f0654709640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:39.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.954+0000 7f0654709640 1 -- 172.21.15.62:0/1625974650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f064c06c990 con 0x7f064c133df0 2024-04-19T01:48:39.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.954+0000 7f063effd640 1 -- 172.21.15.62:0/1625974650 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f0648031e20 con 0x7f064c133df0 2024-04-19T01:48:39.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.954+0000 7f063effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:48:39.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.954+0000 7f063effd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:48:39.956 INFO:teuthology.orchestra.run.smithi062.stdout:{"election_epoch":38,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":94,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","modified":"2024-04-19T01:41:38.497795Z","created":"2024-04-19T01:41:38.497795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:48:39.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.958+0000 7f0654709640 1 -- 172.21.15.62:0/1625974650 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0630036470 msgr2=0x7f0630038930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:39.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.958+0000 7f0654709640 1 --2- 172.21.15.62:0/1625974650 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0630036470 0x7f0630038930 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0644009100 tx=0x7f0644008040 comp rx=0 tx=0).stop 2024-04-19T01:48:39.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.958+0000 7f0654709640 10 monclient: shutdown 2024-04-19T01:48:39.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.958+0000 7f0654709640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:39.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.958+0000 7f0654709640 1 -- 172.21.15.62:0/1625974650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f064c133df0 msgr2=0x7f064c13e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:39.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.958+0000 7f0654709640 1 --2- 172.21.15.62:0/1625974650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f064c133df0 0x7f064c13e290 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f064c0754e0 tx=0x7f0648004bc0 comp rx=0 tx=0).stop 2024-04-19T01:48:39.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.958+0000 7f0654709640 1 -- 172.21.15.62:0/1625974650 shutdown_connections 2024-04-19T01:48:39.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.958+0000 7f0654709640 1 --2- 172.21.15.62:0/1625974650 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0630036470 0x7f0630038930 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:39.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.958+0000 7f0654709640 1 --2- 172.21.15.62:0/1625974650 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f064c133970 0x7f064c13dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:39.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.958+0000 7f0654709640 1 --2- 172.21.15.62:0/1625974650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f064c133df0 0x7f064c13e290 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:39.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.958+0000 7f0654709640 1 --2- 172.21.15.62:0/1625974650 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0638004810 0x7f064c1cf2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:39.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.958+0000 7f0654709640 1 -- 172.21.15.62:0/1625974650 >> 172.21.15.62:0/1625974650 conn(0x7f064c0a7050 msgr2=0x7f064c06f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:39.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.958+0000 7f0654709640 1 -- 172.21.15.62:0/1625974650 shutdown_connections 2024-04-19T01:48:39.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:39.958+0000 7f0654709640 1 -- 172.21.15.62:0/1625974650 wait complete. 2024-04-19T01:48:40.831 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31791]: starting. 2024-04-19T01:48:40.831 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31791]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-31790 2024-04-19T01:48:40.832 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31791]: created object 0... 2024-04-19T01:48:40.832 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31791]: created object 25... 2024-04-19T01:48:40.832 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31791]: created object 49... 2024-04-19T01:48:40.832 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31791]: finishing. 2024-04-19T01:48:40.832 INFO:teuthology.orchestra.run.smithi071.stdout:process_1_[31791]: shutting down. 2024-04-19T01:48:41.531 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[31792]: starting. 2024-04-19T01:48:41.531 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[31792]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-31790 2024-04-19T01:48:41.531 INFO:teuthology.orchestra.run.smithi071.stdout:process_2_[31792]: shutting down. 2024-04-19T01:48:42.976 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:48:43.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.110+0000 7f9c418a4640 10 monclient: get_monmap_and_config 2024-04-19T01:48:43.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.110+0000 7f9c418a4640 10 monclient: build_initial_monmap 2024-04-19T01:48:43.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.110+0000 7f9c418a4640 10 monclient: monmap: 2024-04-19T01:48:43.110 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:43.110 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:43.110 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:48:43.116111+0000 2024-04-19T01:48:43.110 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:48:43.116111+0000 2024-04-19T01:48:43.110 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:43.110 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:43.110 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:48:43.111 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:48:43.111 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:48:43.111 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:48:43.111 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:48:43.111 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:43.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.110+0000 7f9c418a4640 1 Processor -- start 2024-04-19T01:48:43.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.110+0000 7f9c418a4640 1 -- start start 2024-04-19T01:48:43.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.110+0000 7f9c418a4640 10 monclient: init 2024-04-19T01:48:43.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.110+0000 7f9c418a4640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:43.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.110+0000 7f9c418a4640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-04-19T01:48:43.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.110+0000 7f9c418a4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9c3c0a3410 addr v1:172.21.15.71:6789/0 2024-04-19T01:48:43.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.110+0000 7f9c418a4640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9c3c0a2900 0x7f9c3c13c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:43.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.110+0000 7f9c418a4640 10 monclient(hunting): picked mon.noname-c con 0x7f9c3c0a2900 addr v2:172.21.15.71:3300/0 2024-04-19T01:48:43.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.110+0000 7f9c418a4640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9c3c0a2d50 0x7f9c3c13ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:43.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.110+0000 7f9c418a4640 10 monclient(hunting): picked mon.noname-a con 0x7f9c3c0a2d50 addr v2:172.21.15.62:3300/0 2024-04-19T01:48:43.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.110+0000 7f9c418a4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c3c0a56e0 con 0x7f9c3c0a3410 2024-04-19T01:48:43.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.110+0000 7f9c418a4640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:43.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.110+0000 7f9c418a4640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f9c3c0a1430 con 0x7f9c3c0a2d50 2024-04-19T01:48:43.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.110+0000 7f9c418a4640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:43.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.110+0000 7f9c418a4640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f9c3c0a1710 con 0x7f9c3c0a2900 2024-04-19T01:48:43.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.110+0000 7f9c418a4640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:43.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.110+0000 7f9c418a4640 10 monclient(hunting): authenticate will time out at 1308.935639s 2024-04-19T01:48:43.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.110+0000 7f9c3b7fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9c3c0a2d50 0x7f9c3c13ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:43.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.110+0000 7f9c3b7fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9c3c0a2d50 0x7f9c3c13ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39356/0 (socket says 172.21.15.62:39356) 2024-04-19T01:48:43.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.110+0000 7f9c3b7fe640 1 -- 172.21.15.62:0/1404498851 learned_addr learned my addr 172.21.15.62:0/1404498851 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:43.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.110+0000 7f9c3b7fe640 10 monclient(hunting): get_auth_request con 0x7f9c3c0a2d50 auth_method 0 2024-04-19T01:48:43.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.110+0000 7f9c3b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:43.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.110+0000 7f9c3b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:43.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.110+0000 7f9c3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:43.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.114+0000 7f9c3b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:43.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.114+0000 7f9c3b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:43.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.114+0000 7f9c3b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:43.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.114+0000 7f9c3a7fc640 1 --2- 172.21.15.62:0/1404498851 >> v2:172.21.15.71:3300/0 conn(0x7f9c3c0a2900 0x7f9c3c13c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:43.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.114+0000 7f9c3a7fc640 10 monclient(hunting): get_auth_request con 0x7f9c3c0a2900 auth_method 0 2024-04-19T01:48:43.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.114+0000 7f9c3a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:43.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.114+0000 7f9c3a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:43.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.114+0000 7f9c3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:43.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.114+0000 7f9c3b7fe640 10 monclient(hunting): handle_auth_done global_id 15243 payload 274 2024-04-19T01:48:43.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.114+0000 7f9c3b7fe640 1 -- 172.21.15.62:0/1404498851 >> v2:172.21.15.71:3300/0 conn(0x7f9c3c0a2900 msgr2=0x7f9c3c13c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:43.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.114+0000 7f9c3b7fe640 1 --2- 172.21.15.62:0/1404498851 >> v2:172.21.15.71:3300/0 conn(0x7f9c3c0a2900 0x7f9c3c13c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:43.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.114+0000 7f9c3b7fe640 1 -- 172.21.15.62:0/1404498851 >> v1:172.21.15.71:6789/0 conn(0x7f9c3c0a3410 legacy=0x7f9c3c0a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:43.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.114+0000 7f9c3b7fe640 10 monclient: _finish_hunting 0 2024-04-19T01:48:43.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.114+0000 7f9c3b7fe640 1 monclient: found mon.noname-a 2024-04-19T01:48:43.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.114+0000 7f9c3b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:43.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.114+0000 7f9c3b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:48:43.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.114+0000 7f9c3b7fe640 1 -- 172.21.15.62:0/1404498851 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c3c0a19f0 con 0x7f9c3c0a2d50 2024-04-19T01:48:43.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3b7fe640 1 -- 172.21.15.62:0/1404498851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c3c0a2d50 msgr2=0x7f9c3c13ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:43.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3b7fe640 1 --2- 172.21.15.62:0/1404498851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c3c0a2d50 0x7f9c3c13ed00 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c0738e0 tx=0x7f9c3401c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c2afccee3cbc1da server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:43.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c39ffb640 1 -- 172.21.15.62:0/1404498851 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c340a5730 con 0x7f9c3c0a2d50 2024-04-19T01:48:43.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c39ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:43.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c39ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:48:43.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c39ffb640 10 monclient: dump: 2024-04-19T01:48:43.118 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:43.118 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:43.118 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:43.118 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:43.118 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:43.119 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:43.119 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:43.119 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:43.119 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:43.119 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:43.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c39ffb640 10 monclient: _finish_auth 0 2024-04-19T01:48:43.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c39ffb640 10 monclient: _check_auth_tickets 2024-04-19T01:48:43.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c39ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:43.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c39ffb640 1 -- 172.21.15.62:0/1404498851 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c340a58d0 con 0x7f9c3c0a2d50 2024-04-19T01:48:43.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 5 monclient: authenticate success, global_id 15243 2024-04-19T01:48:43.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:48:43.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c39ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:43.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 10 monclient: get_monmap_and_config success 2024-04-19T01:48:43.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c39ffb640 1 -- 172.21.15.62:0/1404498851 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c340a5ca0 con 0x7f9c3c0a2d50 2024-04-19T01:48:43.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c39ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:43.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 4 set_mon_vals no callback set 2024-04-19T01:48:43.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c39ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:48:43.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c39ffb640 10 monclient: dump: 2024-04-19T01:48:43.120 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:43.120 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:43.120 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:43.120 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:43.120 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:43.120 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:43.120 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:43.120 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:43.120 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:43.120 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:43.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 10 monclient: shutdown 2024-04-19T01:48:43.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:43.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 1 -- 172.21.15.62:0/1404498851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c3c0a2d50 msgr2=0x7f9c3c13ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:43.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 1 --2- 172.21.15.62:0/1404498851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c3c0a2d50 0x7f9c3c13ed00 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c0738e0 tx=0x7f9c3401c890 comp rx=0 tx=0).stop 2024-04-19T01:48:43.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 1 -- 172.21.15.62:0/1404498851 shutdown_connections 2024-04-19T01:48:43.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 1 --2- 172.21.15.62:0/1404498851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c3c0a2d50 0x7f9c3c13ed00 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:43.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 1 --2- 172.21.15.62:0/1404498851 >> v2:172.21.15.71:3300/0 conn(0x7f9c3c0a2900 0x7f9c3c13c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:43.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 1 -- 172.21.15.62:0/1404498851 >> 172.21.15.62:0/1404498851 conn(0x7f9c3c0aa2d0 msgr2=0x7f9c3c0a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:43.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 1 -- 172.21.15.62:0/1404498851 shutdown_connections 2024-04-19T01:48:43.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 1 -- 172.21.15.62:0/1404498851 wait complete. 2024-04-19T01:48:43.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 10 monclient: build_initial_monmap 2024-04-19T01:48:43.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 10 monclient: monmap: 2024-04-19T01:48:43.121 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:43.121 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:48:43.121 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:48:43.121 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:48:43.121 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:43.122 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:43.122 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:48:43.122 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:48:43.122 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:48:43.122 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:43.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 1 Processor -- start 2024-04-19T01:48:43.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 1 -- start start 2024-04-19T01:48:43.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 10 monclient: init 2024-04-19T01:48:43.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:43.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-19T01:48:43.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9c3c0a2900 0x7f9c3c1cef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:43.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 10 monclient(hunting): picked mon.noname-c con 0x7f9c3c0a2900 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:43.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c3c0a2d50 0x7f9c3c1cf750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:43.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 10 monclient(hunting): picked mon.noname-a con 0x7f9c3c0a2d50 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:48:43.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c3c0a3410 0x7f9c3c123ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:43.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 10 monclient(hunting): picked mon.noname-b con 0x7f9c3c0a3410 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:48:43.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:43.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9c3c141df0 con 0x7f9c3c0a2d50 2024-04-19T01:48:43.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:43.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f9c3c141c70 con 0x7f9c3c0a3410 2024-04-19T01:48:43.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:43.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f9c3c141f70 con 0x7f9c3c0a2900 2024-04-19T01:48:43.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:43.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c418a4640 10 monclient(hunting): authenticate will time out at 1308.943316s 2024-04-19T01:48:43.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3a7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c3c0a2d50 0x7f9c3c1cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:43.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3affd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9c3c0a2900 0x7f9c3c1cef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:43.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3a7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c3c0a2d50 0x7f9c3c1cf750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39368/0 (socket says 172.21.15.62:39368) 2024-04-19T01:48:43.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3a7fc640 1 -- 172.21.15.62:0/2290216639 learned_addr learned my addr 172.21.15.62:0/2290216639 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:43.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3a7fc640 10 monclient(hunting): get_auth_request con 0x7f9c3c0a2d50 auth_method 0 2024-04-19T01:48:43.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:43.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:43.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:43.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3affd640 10 monclient(hunting): get_auth_request con 0x7f9c3c0a2900 auth_method 0 2024-04-19T01:48:43.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:43.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:43.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:43.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:43.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:43.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:43.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3affd640 1 -- 172.21.15.62:0/2290216639 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9c3c0a2900 msgr2=0x7f9c3c1cef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-19T01:48:43.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3affd640 1 -- 172.21.15.62:0/2290216639 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9c3c0a2900 msgr2=0x7f9c3c1cef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:48:43.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3affd640 1 --2- 172.21.15.62:0/2290216639 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9c3c0a2900 0x7f9c3c1cef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:48:43.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3b7fe640 1 --2- 172.21.15.62:0/2290216639 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c3c0a3410 0x7f9c3c123ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:43.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3affd640 1 --2- 172.21.15.62:0/2290216639 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9c3c0a2900 0x7f9c3c1cef80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:48:43.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3b7fe640 10 monclient(hunting): get_auth_request con 0x7f9c3c0a3410 auth_method 0 2024-04-19T01:48:43.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:43.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:43.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:43.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3a7fc640 10 monclient(hunting): handle_auth_done global_id 15246 payload 824 2024-04-19T01:48:43.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3a7fc640 1 -- 172.21.15.62:0/2290216639 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9c3c0a2900 msgr2=0x7f9c3c1cef80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:48:43.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3a7fc640 1 --2- 172.21.15.62:0/2290216639 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9c3c0a2900 0x7f9c3c1cef80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:43.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3a7fc640 1 -- 172.21.15.62:0/2290216639 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c3c0a3410 msgr2=0x7f9c3c123ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:43.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3a7fc640 1 --2- 172.21.15.62:0/2290216639 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c3c0a3410 0x7f9c3c123ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:43.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3a7fc640 10 monclient: _finish_hunting 0 2024-04-19T01:48:43.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3a7fc640 1 monclient: found mon.noname-a 2024-04-19T01:48:43.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:43.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:48:43.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.118+0000 7f9c3a7fc640 1 -- 172.21.15.62:0/2290216639 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c3c127f40 con 0x7f9c3c0a2d50 2024-04-19T01:48:43.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.122+0000 7f9c3a7fc640 1 -- 172.21.15.62:0/2290216639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c3c0a2d50 msgr2=0x7f9c3c1cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:43.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.122+0000 7f9c3a7fc640 1 --2- 172.21.15.62:0/2290216639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c3c0a2d50 0x7f9c3c1cf750 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f9c2400d800 tx=0x7f9c2400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:43.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.122+0000 7f9c33fff640 1 -- 172.21.15.62:0/2290216639 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c2400b870 con 0x7f9c3c0a2d50 2024-04-19T01:48:43.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.122+0000 7f9c33fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:43.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.122+0000 7f9c33fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:48:43.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.122+0000 7f9c33fff640 10 monclient: dump: 2024-04-19T01:48:43.130 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:43.130 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:43.130 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:43.131 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:43.131 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:43.131 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:43.131 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:43.131 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:43.131 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:43.131 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:43.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.122+0000 7f9c33fff640 10 monclient: _finish_auth 0 2024-04-19T01:48:43.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.122+0000 7f9c33fff640 10 monclient: _check_auth_tickets 2024-04-19T01:48:43.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.122+0000 7f9c33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:43.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.122+0000 7f9c33fff640 1 -- 172.21.15.62:0/2290216639 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c24019070 con 0x7f9c3c0a2d50 2024-04-19T01:48:43.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.122+0000 7f9c33fff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:43.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.122+0000 7f9c33fff640 1 -- 172.21.15.62:0/2290216639 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c2400bc10 con 0x7f9c3c0a2d50 2024-04-19T01:48:43.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.122+0000 7f9c33fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:43.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.122+0000 7f9c33fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:48:43.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.122+0000 7f9c33fff640 10 monclient: dump: 2024-04-19T01:48:43.132 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:43.132 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:43.132 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:43.132 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:43.132 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:43.132 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:43.132 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:43.132 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:43.132 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:43.132 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:43.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.122+0000 7f9c397fa640 4 set_mon_vals no callback set 2024-04-19T01:48:43.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.122+0000 7f9c418a4640 5 monclient: authenticate success, global_id 15246 2024-04-19T01:48:43.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.122+0000 7f9c418a4640 10 monclient: _renew_subs 2024-04-19T01:48:43.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.122+0000 7f9c418a4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:43.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.122+0000 7f9c418a4640 1 -- 172.21.15.62:0/2290216639 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c3c0a3f80 con 0x7f9c3c0a2d50 2024-04-19T01:48:43.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.122+0000 7f9c418a4640 10 monclient: _renew_subs 2024-04-19T01:48:43.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.122+0000 7f9c418a4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:43.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.122+0000 7f9c418a4640 1 -- 172.21.15.62:0/2290216639 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c3c1276a0 con 0x7f9c3c0a2d50 2024-04-19T01:48:43.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.122+0000 7f9c418a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:43.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.122+0000 7f9c418a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:43.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.122+0000 7f9c418a4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:43.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.122+0000 7f9c33fff640 1 -- 172.21.15.62:0/2290216639 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f9c2401e460 con 0x7f9c3c0a2d50 2024-04-19T01:48:43.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.122+0000 7f9c418a4640 1 -- 172.21.15.62:0/2290216639 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c3c0692b0 con 0x7f9c3c0a2d50 2024-04-19T01:48:43.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.126+0000 7f9c33fff640 1 --2- 172.21.15.62:0/2290216639 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f9c1c036420 0x7f9c1c0388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:43.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.126+0000 7f9c33fff640 1 -- 172.21.15.62:0/2290216639 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(322..322 src has 115..322) ==== 4738+0+0 (secure 0 0 0) 0x7f9c24049af0 con 0x7f9c3c0a2d50 2024-04-19T01:48:43.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.126+0000 7f9c3affd640 1 --2- 172.21.15.62:0/2290216639 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f9c1c036420 0x7f9c1c0388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:43.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.126+0000 7f9c3affd640 10 monclient: get_auth_request con 0x7f9c1c036420 auth_method 0 2024-04-19T01:48:43.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.126+0000 7f9c3affd640 1 --2- 172.21.15.62:0/2290216639 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f9c1c036420 0x7f9c1c0388e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c007fd0 tx=0x7f9c2c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:43.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.130+0000 7f9c33fff640 1 -- 172.21.15.62:0/2290216639 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9c24048510 con 0x7f9c3c0a2d50 2024-04-19T01:48:43.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.130+0000 7f9c33fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:43.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.130+0000 7f9c33fff640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:48:43.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.354+0000 7f9c418a4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:48:43.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.354+0000 7f9c418a4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:48:43.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.354+0000 7f9c418a4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:43.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.354+0000 7f9c418a4640 1 -- 172.21.15.62:0/2290216639 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9c3c06b980 con 0x7f9c3c0a2d50 2024-04-19T01:48:43.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.354+0000 7f9c33fff640 1 -- 172.21.15.62:0/2290216639 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f9c24014430 con 0x7f9c3c0a2d50 2024-04-19T01:48:43.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.354+0000 7f9c33fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:48:43.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.354+0000 7f9c33fff640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:48:43.353 INFO:teuthology.orchestra.run.smithi062.stdout:{"election_epoch":38,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":97,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","modified":"2024-04-19T01:41:38.497795Z","created":"2024-04-19T01:41:38.497795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:48:43.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.354+0000 7f9c418a4640 1 -- 172.21.15.62:0/2290216639 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f9c1c036420 msgr2=0x7f9c1c0388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:43.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.354+0000 7f9c418a4640 1 --2- 172.21.15.62:0/2290216639 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f9c1c036420 0x7f9c1c0388e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c007fd0 tx=0x7f9c2c009040 comp rx=0 tx=0).stop 2024-04-19T01:48:43.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.354+0000 7f9c418a4640 10 monclient: shutdown 2024-04-19T01:48:43.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.354+0000 7f9c418a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:43.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.354+0000 7f9c418a4640 1 -- 172.21.15.62:0/2290216639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c3c0a2d50 msgr2=0x7f9c3c1cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:43.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.354+0000 7f9c418a4640 1 --2- 172.21.15.62:0/2290216639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c3c0a2d50 0x7f9c3c1cf750 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f9c2400d800 tx=0x7f9c2400dcc0 comp rx=0 tx=0).stop 2024-04-19T01:48:43.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.354+0000 7f9c418a4640 1 -- 172.21.15.62:0/2290216639 shutdown_connections 2024-04-19T01:48:43.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.354+0000 7f9c418a4640 1 --2- 172.21.15.62:0/2290216639 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c3c0a3410 0x7f9c3c123ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:43.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.354+0000 7f9c418a4640 1 --2- 172.21.15.62:0/2290216639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c3c0a2d50 0x7f9c3c1cf750 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:43.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.354+0000 7f9c418a4640 1 --2- 172.21.15.62:0/2290216639 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f9c1c036420 0x7f9c1c0388e0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:43.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.354+0000 7f9c418a4640 1 --2- 172.21.15.62:0/2290216639 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f9c3c0a2900 0x7f9c3c1cef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:43.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.354+0000 7f9c418a4640 1 -- 172.21.15.62:0/2290216639 >> 172.21.15.62:0/2290216639 conn(0x7f9c3c0aa2d0 msgr2=0x7f9c3c0a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:43.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.354+0000 7f9c418a4640 1 -- 172.21.15.62:0/2290216639 shutdown_connections 2024-04-19T01:48:43.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:43.354+0000 7f9c418a4640 1 -- 172.21.15.62:0/2290216639 wait complete. 2024-04-19T01:48:43.814 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:43.815 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31825]: starting. 2024-04-19T01:48:43.815 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31825]: creating pool ceph_test_rados_delete_pools_parallel.smithi071-31790 2024-04-19T01:48:43.815 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31825]: created object 0... 2024-04-19T01:48:43.815 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31825]: created object 25... 2024-04-19T01:48:43.815 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31825]: created object 49... 2024-04-19T01:48:43.815 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31825]: finishing. 2024-04-19T01:48:43.815 INFO:teuthology.orchestra.run.smithi071.stdout:process_3_[31825]: shutting down. 2024-04-19T01:48:43.875 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:43.875 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31827]: starting. 2024-04-19T01:48:43.875 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31827]: listing objects. 2024-04-19T01:48:43.875 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31827]: listed object 0... 2024-04-19T01:48:43.876 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31827]: listed object 25... 2024-04-19T01:48:43.876 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31827]: saw 50 objects 2024-04-19T01:48:43.876 INFO:teuthology.orchestra.run.smithi071.stdout:process_5_[31827]: shutting down. 2024-04-19T01:48:44.620 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:44.620 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[31826]: starting. 2024-04-19T01:48:44.620 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[31826]: deleting pool ceph_test_rados_delete_pools_parallel.smithi071-31790 2024-04-19T01:48:44.620 INFO:teuthology.orchestra.run.smithi071.stdout:process_4_[31826]: shutting down. 2024-04-19T01:48:44.624 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:44.624 INFO:teuthology.orchestra.run.smithi071.stdout:******************************* 2024-04-19T01:48:44.624 INFO:teuthology.orchestra.run.smithi071.stdout:******* SUCCESS ********** 2024-04-19T01:48:44.627 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-04-19T01:48:44.637 INFO:tasks.mon_thrash:joining mon_thrasher 2024-04-19T01:48:46.372 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:48:46.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a72452640 10 monclient: get_monmap_and_config 2024-04-19T01:48:46.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a72452640 10 monclient: build_initial_monmap 2024-04-19T01:48:46.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a72452640 10 monclient: monmap: 2024-04-19T01:48:46.506 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:46.506 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:46.506 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:48:46.511706+0000 2024-04-19T01:48:46.506 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:48:46.511706+0000 2024-04-19T01:48:46.506 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:46.506 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:46.506 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:48:46.506 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:48:46.506 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:48:46.506 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:48:46.506 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:48:46.507 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:46.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a72452640 1 Processor -- start 2024-04-19T01:48:46.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a72452640 1 -- start start 2024-04-19T01:48:46.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a72452640 10 monclient: init 2024-04-19T01:48:46.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a72452640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:46.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a72452640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-04-19T01:48:46.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a72452640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a6c0a47e0 0x7f4a6c09f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:46.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a72452640 10 monclient(hunting): picked mon.noname-b con 0x7f4a6c0a47e0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:46.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a72452640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4a6c09f890 addr v1:172.21.15.62:6789/0 2024-04-19T01:48:46.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a72452640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4a6c0a52d0 0x7f4a6c13b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:46.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a72452640 10 monclient(hunting): picked mon.noname-c con 0x7f4a6c0a52d0 addr v2:172.21.15.71:3300/0 2024-04-19T01:48:46.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a72452640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a6c09ec50 con 0x7f4a6c09f890 2024-04-19T01:48:46.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a72452640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:46.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a72452640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f4a6c09ef30 con 0x7f4a6c0a52d0 2024-04-19T01:48:46.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a72452640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:46.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a72452640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f4a6c09e690 con 0x7f4a6c0a47e0 2024-04-19T01:48:46.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a72452640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:46.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a72452640 10 monclient(hunting): authenticate will time out at 1312.331097s 2024-04-19T01:48:46.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a6b7fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f4a6c09f890 0x7f4a6c0a4bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:56660/0 (socket says 172.21.15.62:56660) 2024-04-19T01:48:46.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a6bfff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a6c0a47e0 0x7f4a6c09f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:46.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a6b7fe640 1 -- 172.21.15.62:0/2431091201 learned_addr learned my addr 172.21.15.62:0/2431091201 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:48:46.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a6bfff640 10 monclient(hunting): get_auth_request con 0x7f4a6c0a47e0 auth_method 0 2024-04-19T01:48:46.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:46.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a6bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:46.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:46.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a709ef640 1 --2- 172.21.15.62:0/2431091201 >> v2:172.21.15.71:3300/0 conn(0x7f4a6c0a52d0 0x7f4a6c13b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:46.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a6bfff640 1 -- 172.21.15.62:0/2431091201 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a6c0a47e0 msgr2=0x7f4a6c09f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:48:46.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a6bfff640 1 -- 172.21.15.62:0/2431091201 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a6c0a47e0 msgr2=0x7f4a6c09f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:48:46.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a6bfff640 1 --2- 172.21.15.62:0/2431091201 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a6c0a47e0 0x7f4a6c09f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:48:46.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a709ef640 10 monclient(hunting): get_auth_request con 0x7f4a6c0a52d0 auth_method 0 2024-04-19T01:48:46.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a709ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:46.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a709ef640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:46.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a709ef640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:46.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a6bfff640 1 --2- 172.21.15.62:0/2431091201 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a6c0a47e0 0x7f4a6c09f350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:48:46.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a709ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:46.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a709ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:46.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.506+0000 7f4a709ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:46.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.510+0000 7f4a709ef640 10 monclient(hunting): handle_auth_done global_id 24505 payload 274 2024-04-19T01:48:46.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.510+0000 7f4a709ef640 1 -- 172.21.15.62:0/2431091201 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a6c0a47e0 msgr2=0x7f4a6c09f350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:48:46.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.510+0000 7f4a709ef640 1 --2- 172.21.15.62:0/2431091201 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a6c0a47e0 0x7f4a6c09f350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:46.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.510+0000 7f4a709ef640 1 -- 172.21.15.62:0/2431091201 >> v1:172.21.15.62:6789/0 conn(0x7f4a6c09f890 legacy=0x7f4a6c0a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:46.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.510+0000 7f4a709ef640 10 monclient: _finish_hunting 0 2024-04-19T01:48:46.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.510+0000 7f4a709ef640 1 monclient: found mon.noname-c 2024-04-19T01:48:46.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.510+0000 7f4a709ef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:46.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.510+0000 7f4a709ef640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-04-19T01:48:46.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.510+0000 7f4a709ef640 1 -- 172.21.15.62:0/2431091201 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a6c09e970 con 0x7f4a6c0a52d0 2024-04-19T01:48:46.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a709ef640 1 -- 172.21.15.62:0/2431091201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a6c0a52d0 msgr2=0x7f4a6c13b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:46.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a709ef640 1 --2- 172.21.15.62:0/2431091201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a6c0a52d0 0x7f4a6c13b570 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4a640097c0 tx=0x7f4a6401c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcec7b8f1f99fb8e server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:46.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a6affd640 1 -- 172.21.15.62:0/2431091201 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a640b2070 con 0x7f4a6c0a52d0 2024-04-19T01:48:46.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:46.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a6affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:48:46.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a6affd640 10 monclient: dump: 2024-04-19T01:48:46.518 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:46.518 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:46.518 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:46.518 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:46.518 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:46.518 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:46.518 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:46.518 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:46.519 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:46.519 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:46.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a6affd640 10 monclient: _finish_auth 0 2024-04-19T01:48:46.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a6affd640 10 monclient: _check_auth_tickets 2024-04-19T01:48:46.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:46.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a6affd640 1 -- 172.21.15.62:0/2431091201 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a640ad640 con 0x7f4a6c0a52d0 2024-04-19T01:48:46.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a6affd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:46.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a6affd640 1 -- 172.21.15.62:0/2431091201 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a640ada10 con 0x7f4a6c0a52d0 2024-04-19T01:48:46.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 5 monclient: authenticate success, global_id 24505 2024-04-19T01:48:46.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 10 monclient: get_monmap_and_config success 2024-04-19T01:48:46.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 4 set_mon_vals no callback set 2024-04-19T01:48:46.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:46.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a6affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:48:46.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a6affd640 10 monclient: dump: 2024-04-19T01:48:46.520 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:46.520 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:46.520 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:46.520 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:46.520 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:46.520 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:46.520 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:46.520 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:46.520 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:46.520 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:46.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 10 monclient: shutdown 2024-04-19T01:48:46.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:46.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 1 -- 172.21.15.62:0/2431091201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a6c0a52d0 msgr2=0x7f4a6c13b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:46.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 1 --2- 172.21.15.62:0/2431091201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a6c0a52d0 0x7f4a6c13b570 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4a640097c0 tx=0x7f4a6401c3f0 comp rx=0 tx=0).stop 2024-04-19T01:48:46.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 1 -- 172.21.15.62:0/2431091201 shutdown_connections 2024-04-19T01:48:46.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 1 --2- 172.21.15.62:0/2431091201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a6c0a52d0 0x7f4a6c13b570 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:46.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 1 --2- 172.21.15.62:0/2431091201 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a6c0a47e0 0x7f4a6c09f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:46.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 1 -- 172.21.15.62:0/2431091201 >> 172.21.15.62:0/2431091201 conn(0x7f4a6c0a23d0 msgr2=0x7f4a6c0a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:46.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 1 -- 172.21.15.62:0/2431091201 shutdown_connections 2024-04-19T01:48:46.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 1 -- 172.21.15.62:0/2431091201 wait complete. 2024-04-19T01:48:46.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 10 monclient: build_initial_monmap 2024-04-19T01:48:46.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 10 monclient: monmap: 2024-04-19T01:48:46.521 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:46.521 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:48:46.521 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:48:46.521 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:48:46.522 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:46.522 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:46.522 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:48:46.522 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:48:46.522 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:48:46.522 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:46.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 1 Processor -- start 2024-04-19T01:48:46.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 1 -- start start 2024-04-19T01:48:46.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 10 monclient: init 2024-04-19T01:48:46.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:46.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-19T01:48:46.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a6c09f890 0x7f4a6c137380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:46.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 10 monclient(hunting): picked mon.noname-a con 0x7f4a6c09f890 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:48:46.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a6c0a47e0 0x7f4a6c11ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:46.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 10 monclient(hunting): picked mon.noname-b con 0x7f4a6c0a47e0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:48:46.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a6c0a52d0 0x7f4a6c11f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:46.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 10 monclient(hunting): picked mon.noname-c con 0x7f4a6c0a52d0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:46.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:46.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4a6c13e800 con 0x7f4a6c09f890 2024-04-19T01:48:46.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:46.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4a6c13e680 con 0x7f4a6c0a47e0 2024-04-19T01:48:46.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:46.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f4a6c13e980 con 0x7f4a6c0a52d0 2024-04-19T01:48:46.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:46.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a72452640 10 monclient(hunting): authenticate will time out at 1312.339767s 2024-04-19T01:48:46.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a6bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a6c09f890 0x7f4a6c137380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:46.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a6bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a6c09f890 0x7f4a6c137380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47684/0 (socket says 172.21.15.62:47684) 2024-04-19T01:48:46.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a6bfff640 1 -- 172.21.15.62:0/4140086552 learned_addr learned my addr 172.21.15.62:0/4140086552 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:46.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a6bfff640 10 monclient(hunting): get_auth_request con 0x7f4a6c09f890 auth_method 0 2024-04-19T01:48:46.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:46.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a6bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:46.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:46.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a709ef640 1 --2- 172.21.15.62:0/4140086552 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a6c0a52d0 0x7f4a6c11f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:46.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a709ef640 10 monclient(hunting): get_auth_request con 0x7f4a6c0a52d0 auth_method 0 2024-04-19T01:48:46.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a709ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:46.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a709ef640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:46.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.514+0000 7f4a709ef640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:46.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:46.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:46.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:46.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a709ef640 1 -- 172.21.15.62:0/4140086552 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a6c0a52d0 msgr2=0x7f4a6c11f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-19T01:48:46.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a709ef640 1 -- 172.21.15.62:0/4140086552 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a6c0a52d0 msgr2=0x7f4a6c11f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:48:46.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a709ef640 1 --2- 172.21.15.62:0/4140086552 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a6c0a52d0 0x7f4a6c11f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:48:46.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a709ef640 1 --2- 172.21.15.62:0/4140086552 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a6c0a52d0 0x7f4a6c11f170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:48:46.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a6b7fe640 1 --2- 172.21.15.62:0/4140086552 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a6c0a47e0 0x7f4a6c11ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:46.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a6b7fe640 10 monclient(hunting): get_auth_request con 0x7f4a6c0a47e0 auth_method 0 2024-04-19T01:48:46.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:46.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a6b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:46.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:46.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a6bfff640 10 monclient(hunting): handle_auth_done global_id 15249 payload 824 2024-04-19T01:48:46.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a6bfff640 1 -- 172.21.15.62:0/4140086552 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a6c0a52d0 msgr2=0x7f4a6c11f170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:48:46.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a6bfff640 1 --2- 172.21.15.62:0/4140086552 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a6c0a52d0 0x7f4a6c11f170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:46.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a6bfff640 1 -- 172.21.15.62:0/4140086552 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a6c0a47e0 msgr2=0x7f4a6c11ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:46.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a6bfff640 1 --2- 172.21.15.62:0/4140086552 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a6c0a47e0 0x7f4a6c11ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:46.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a6bfff640 10 monclient: _finish_hunting 0 2024-04-19T01:48:46.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a6bfff640 1 monclient: found mon.noname-a 2024-04-19T01:48:46.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:46.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a6bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:48:46.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a6bfff640 1 -- 172.21.15.62:0/4140086552 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a6c1cb1e0 con 0x7f4a6c09f890 2024-04-19T01:48:46.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a6bfff640 1 -- 172.21.15.62:0/4140086552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a6c09f890 msgr2=0x7f4a6c137380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:46.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a6bfff640 1 --2- 172.21.15.62:0/4140086552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a6c09f890 0x7f4a6c137380 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c007ef0 tx=0x7f4a5c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:46.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a68ff9640 1 -- 172.21.15.62:0/4140086552 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c013790 con 0x7f4a6c09f890 2024-04-19T01:48:46.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a68ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:46.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a68ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:48:46.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a68ff9640 10 monclient: dump: 2024-04-19T01:48:46.532 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:46.533 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:46.533 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:46.533 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:46.533 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:46.533 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:46.533 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:46.533 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:46.533 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:46.534 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:46.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a68ff9640 10 monclient: _finish_auth 0 2024-04-19T01:48:46.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a68ff9640 10 monclient: _check_auth_tickets 2024-04-19T01:48:46.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a68ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:46.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a68ff9640 1 -- 172.21.15.62:0/4140086552 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a5c019070 con 0x7f4a6c09f890 2024-04-19T01:48:46.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a72452640 5 monclient: authenticate success, global_id 15249 2024-04-19T01:48:46.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a72452640 10 monclient: _renew_subs 2024-04-19T01:48:46.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a72452640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:46.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a72452640 1 -- 172.21.15.62:0/4140086552 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a6c122320 con 0x7f4a6c09f890 2024-04-19T01:48:46.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a68ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:46.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a68ff9640 1 -- 172.21.15.62:0/4140086552 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c013b70 con 0x7f4a6c09f890 2024-04-19T01:48:46.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a68ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:46.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a68ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:48:46.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a68ff9640 10 monclient: dump: 2024-04-19T01:48:46.535 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:46.535 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:46.535 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:46.535 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:46.535 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:46.535 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:46.535 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:46.535 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:46.535 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:46.535 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:46.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a6a7fc640 4 set_mon_vals no callback set 2024-04-19T01:48:46.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a72452640 10 monclient: _renew_subs 2024-04-19T01:48:46.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a72452640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:46.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a72452640 1 -- 172.21.15.62:0/4140086552 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a6c1208d0 con 0x7f4a6c09f890 2024-04-19T01:48:46.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a72452640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:46.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a72452640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:46.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a72452640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:46.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a72452640 1 -- 172.21.15.62:0/4140086552 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a6c0692b0 con 0x7f4a6c09f890 2024-04-19T01:48:46.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.518+0000 7f4a68ff9640 1 -- 172.21.15.62:0/4140086552 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f4a5c013d10 con 0x7f4a6c09f890 2024-04-19T01:48:46.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.522+0000 7f4a68ff9640 1 --2- 172.21.15.62:0/4140086552 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4a580364a0 0x7f4a58038960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:46.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.522+0000 7f4a68ff9640 1 -- 172.21.15.62:0/4140086552 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(324..324 src has 115..324) ==== 4346+0+0 (secure 0 0 0) 0x7f4a5c049c60 con 0x7f4a6c09f890 2024-04-19T01:48:46.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.522+0000 7f4a6b7fe640 1 --2- 172.21.15.62:0/4140086552 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4a580364a0 0x7f4a58038960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:46.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.522+0000 7f4a6b7fe640 10 monclient: get_auth_request con 0x7f4a580364a0 auth_method 0 2024-04-19T01:48:46.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.522+0000 7f4a6b7fe640 1 --2- 172.21.15.62:0/4140086552 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4a580364a0 0x7f4a58038960 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4a60007fd0 tx=0x7f4a60009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:46.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.526+0000 7f4a68ff9640 1 -- 172.21.15.62:0/4140086552 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4a6c122320 con 0x7f4a6c09f890 2024-04-19T01:48:46.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.526+0000 7f4a68ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:46.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.526+0000 7f4a68ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:48:46.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.758+0000 7f4a72452640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:48:46.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.758+0000 7f4a72452640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:48:46.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.758+0000 7f4a72452640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:46.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.758+0000 7f4a72452640 1 -- 172.21.15.62:0/4140086552 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4a6c06b980 con 0x7f4a6c09f890 2024-04-19T01:48:46.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.758+0000 7f4a68ff9640 1 -- 172.21.15.62:0/4140086552 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f4a6c1cb1e0 con 0x7f4a6c09f890 2024-04-19T01:48:46.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.758+0000 7f4a68ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:48:46.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.758+0000 7f4a68ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:48:46.760 INFO:teuthology.orchestra.run.smithi062.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","modified":"2024-04-19T01:41:38.497795Z","created":"2024-04-19T01:41:38.497795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:48:46.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.762+0000 7f4a72452640 1 -- 172.21.15.62:0/4140086552 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4a580364a0 msgr2=0x7f4a58038960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:46.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.762+0000 7f4a72452640 1 --2- 172.21.15.62:0/4140086552 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4a580364a0 0x7f4a58038960 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4a60007fd0 tx=0x7f4a60009040 comp rx=0 tx=0).stop 2024-04-19T01:48:46.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.762+0000 7f4a72452640 10 monclient: shutdown 2024-04-19T01:48:46.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.762+0000 7f4a72452640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:46.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.762+0000 7f4a72452640 1 -- 172.21.15.62:0/4140086552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a6c09f890 msgr2=0x7f4a6c137380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:46.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.762+0000 7f4a72452640 1 --2- 172.21.15.62:0/4140086552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a6c09f890 0x7f4a6c137380 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c007ef0 tx=0x7f4a5c00e4b0 comp rx=0 tx=0).stop 2024-04-19T01:48:46.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.762+0000 7f4a72452640 1 -- 172.21.15.62:0/4140086552 shutdown_connections 2024-04-19T01:48:46.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.762+0000 7f4a72452640 1 --2- 172.21.15.62:0/4140086552 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4a580364a0 0x7f4a58038960 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:46.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.762+0000 7f4a72452640 1 --2- 172.21.15.62:0/4140086552 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4a6c0a52d0 0x7f4a6c11f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:46.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.762+0000 7f4a72452640 1 --2- 172.21.15.62:0/4140086552 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a6c0a47e0 0x7f4a6c11ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:46.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.762+0000 7f4a72452640 1 --2- 172.21.15.62:0/4140086552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a6c09f890 0x7f4a6c137380 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:46.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.762+0000 7f4a72452640 1 -- 172.21.15.62:0/4140086552 >> 172.21.15.62:0/4140086552 conn(0x7f4a6c0a23d0 msgr2=0x7f4a6c138480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:46.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.762+0000 7f4a72452640 1 -- 172.21.15.62:0/4140086552 shutdown_connections 2024-04-19T01:48:46.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.762+0000 7f4a72452640 1 -- 172.21.15.62:0/4140086552 wait complete. 2024-04-19T01:48:46.779 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-19T01:48:46.779 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-19T01:48:46.779 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-19T01:48:46.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24b115c640 10 monclient: get_monmap_and_config 2024-04-19T01:48:46.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24b115c640 10 monclient: build_initial_monmap 2024-04-19T01:48:46.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24b115c640 10 monclient: monmap: 2024-04-19T01:48:46.908 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:46.908 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:46.908 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:48:46.914339+0000 2024-04-19T01:48:46.908 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:48:46.914339+0000 2024-04-19T01:48:46.908 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:46.909 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:46.909 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:48:46.909 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:48:46.909 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:48:46.909 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:48:46.909 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:48:46.909 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:46.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24b115c640 1 Processor -- start 2024-04-19T01:48:46.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24b115c640 1 -- start start 2024-04-19T01:48:46.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24b115c640 10 monclient: init 2024-04-19T01:48:46.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24b115c640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:46.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24b115c640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-04-19T01:48:46.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24b115c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24ac0a45d0 0x7f24ac09c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:46.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24b115c640 10 monclient(hunting): picked mon.noname-b con 0x7f24ac0a45d0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:46.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24b115c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f24ac0a49b0 0x7f24ac0a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:46.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24b115c640 10 monclient(hunting): picked mon.noname-a con 0x7f24ac0a49b0 addr v2:172.21.15.62:3300/0 2024-04-19T01:48:46.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24b115c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f24ac137390 addr v1:172.21.15.71:6789/0 2024-04-19T01:48:46.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24b115c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24ac09bc80 con 0x7f24ac137390 2024-04-19T01:48:46.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24b115c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:46.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24b115c640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f24ac09c240 con 0x7f24ac0a49b0 2024-04-19T01:48:46.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24b115c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:46.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24b115c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f24ac09c520 con 0x7f24ac0a45d0 2024-04-19T01:48:46.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24b115c640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:46.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24b115c640 10 monclient(hunting): authenticate will time out at 1312.733846s 2024-04-19T01:48:46.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24aad9d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24ac0a45d0 0x7f24ac09c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:46.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24aad9d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24ac0a45d0 0x7f24ac09c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:58000/0 (socket says 172.21.15.62:58000) 2024-04-19T01:48:46.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24aad9d640 1 -- 172.21.15.62:0/2639196189 learned_addr learned my addr 172.21.15.62:0/2639196189 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:46.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24aad9d640 10 monclient(hunting): get_auth_request con 0x7f24ac0a45d0 auth_method 0 2024-04-19T01:48:46.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:46.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24aad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:46.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24aad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:46.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24a259c640 1 --2- 172.21.15.62:0/2639196189 >> v2:172.21.15.62:3300/0 conn(0x7f24ac0a49b0 0x7f24ac0a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:46.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24a259c640 10 monclient(hunting): get_auth_request con 0x7f24ac0a49b0 auth_method 0 2024-04-19T01:48:46.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:46.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24a259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:46.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24a259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:46.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24aad9d640 1 -- 172.21.15.62:0/2639196189 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24ac0a45d0 msgr2=0x7f24ac09c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:48:46.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24aad9d640 1 -- 172.21.15.62:0/2639196189 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24ac0a45d0 msgr2=0x7f24ac09c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:48:46.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24aad9d640 1 --2- 172.21.15.62:0/2639196189 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24ac0a45d0 0x7f24ac09c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:48:46.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24aad9d640 1 --2- 172.21.15.62:0/2639196189 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24ac0a45d0 0x7f24ac09c940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:48:46.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24a259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:46.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24a259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:46.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24a259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:46.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24a259c640 10 monclient(hunting): handle_auth_done global_id 15252 payload 274 2024-04-19T01:48:46.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24a259c640 1 -- 172.21.15.62:0/2639196189 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24ac0a45d0 msgr2=0x7f24ac09c940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:48:46.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24a259c640 1 --2- 172.21.15.62:0/2639196189 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24ac0a45d0 0x7f24ac09c940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:46.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24a259c640 1 -- 172.21.15.62:0/2639196189 >> v1:172.21.15.71:6789/0 conn(0x7f24ac137390 legacy=0x7f24ac1377e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:46.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24a259c640 10 monclient: _finish_hunting 0 2024-04-19T01:48:46.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24a259c640 1 monclient: found mon.noname-a 2024-04-19T01:48:46.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24a259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:46.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24a259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:48:46.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.910+0000 7f24a259c640 1 -- 172.21.15.62:0/2639196189 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24ac0a61e0 con 0x7f24ac0a49b0 2024-04-19T01:48:46.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24a259c640 1 -- 172.21.15.62:0/2639196189 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ac0a49b0 msgr2=0x7f24ac0a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:46.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24a259c640 1 --2- 172.21.15.62:0/2639196189 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ac0a49b0 0x7f24ac0a4d90 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f24940097c0 tx=0x7f249401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e456471b27571b28 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:46.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24aa59c640 1 -- 172.21.15.62:0/2639196189 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24940b0070 con 0x7f24ac0a49b0 2024-04-19T01:48:46.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24aa59c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:46.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24aa59c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:48:46.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24aa59c640 10 monclient: dump: 2024-04-19T01:48:46.917 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:46.917 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:46.917 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:46.917 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:46.917 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:46.917 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:46.917 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:46.917 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:46.917 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:46.917 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:46.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24aa59c640 10 monclient: _finish_auth 0 2024-04-19T01:48:46.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24aa59c640 10 monclient: _check_auth_tickets 2024-04-19T01:48:46.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24aa59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:46.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24aa59c640 1 -- 172.21.15.62:0/2639196189 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24940ab400 con 0x7f24ac0a49b0 2024-04-19T01:48:46.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24aa59c640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:46.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24aa59c640 1 -- 172.21.15.62:0/2639196189 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24940ab7d0 con 0x7f24ac0a49b0 2024-04-19T01:48:46.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 5 monclient: authenticate success, global_id 15252 2024-04-19T01:48:46.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 10 monclient: get_monmap_and_config success 2024-04-19T01:48:46.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 4 set_mon_vals no callback set 2024-04-19T01:48:46.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24aa59c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:46.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24aa59c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:48:46.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24aa59c640 10 monclient: dump: 2024-04-19T01:48:46.918 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:46.918 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:46.919 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:46.919 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:46.919 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:46.919 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:46.919 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:46.919 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:46.919 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:46.919 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:46.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 10 monclient: shutdown 2024-04-19T01:48:46.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:46.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 1 -- 172.21.15.62:0/2639196189 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ac0a49b0 msgr2=0x7f24ac0a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:46.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 1 --2- 172.21.15.62:0/2639196189 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ac0a49b0 0x7f24ac0a4d90 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f24940097c0 tx=0x7f249401c3f0 comp rx=0 tx=0).stop 2024-04-19T01:48:46.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 1 -- 172.21.15.62:0/2639196189 shutdown_connections 2024-04-19T01:48:46.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 1 --2- 172.21.15.62:0/2639196189 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ac0a49b0 0x7f24ac0a4d90 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:46.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 1 --2- 172.21.15.62:0/2639196189 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24ac0a45d0 0x7f24ac09c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:46.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 1 -- 172.21.15.62:0/2639196189 >> 172.21.15.62:0/2639196189 conn(0x7f24ac0aa1d0 msgr2=0x7f24ac0a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:46.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 1 -- 172.21.15.62:0/2639196189 shutdown_connections 2024-04-19T01:48:46.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 1 -- 172.21.15.62:0/2639196189 wait complete. 2024-04-19T01:48:46.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 10 monclient: build_initial_monmap 2024-04-19T01:48:46.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 10 monclient: monmap: 2024-04-19T01:48:46.926 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:46.926 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:48:46.926 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:48:46.926 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:48:46.926 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:46.926 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:46.926 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:48:46.926 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:48:46.926 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:48:46.926 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:46.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 1 Processor -- start 2024-04-19T01:48:46.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 1 -- start start 2024-04-19T01:48:46.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 10 monclient: init 2024-04-19T01:48:46.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:46.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-19T01:48:46.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24ac0a45d0 0x7f24ac13eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:46.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 10 monclient(hunting): picked mon.noname-c con 0x7f24ac0a45d0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:46.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ac0a49b0 0x7f24ac1363d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:46.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 10 monclient(hunting): picked mon.noname-a con 0x7f24ac0a49b0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:48:46.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f24ac137390 0x7f24ac1cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:46.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 10 monclient(hunting): picked mon.noname-b con 0x7f24ac137390 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:48:46.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:46.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f24ac142310 con 0x7f24ac0a49b0 2024-04-19T01:48:46.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:46.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f24ac138300 con 0x7f24ac137390 2024-04-19T01:48:46.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:46.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f24ac142490 con 0x7f24ac0a45d0 2024-04-19T01:48:46.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:46.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24b115c640 10 monclient(hunting): authenticate will time out at 1312.743642s 2024-04-19T01:48:46.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24aad9d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24ac0a45d0 0x7f24ac13eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:46.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24aad9d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24ac0a45d0 0x7f24ac13eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:58016/0 (socket says 172.21.15.62:58016) 2024-04-19T01:48:46.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24a259c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ac0a49b0 0x7f24ac1363d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:46.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24aad9d640 1 -- 172.21.15.62:0/2987512634 learned_addr learned my addr 172.21.15.62:0/2987512634 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:46.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24aad9d640 10 monclient(hunting): get_auth_request con 0x7f24ac0a45d0 auth_method 0 2024-04-19T01:48:46.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:46.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24aad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:46.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24aad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:46.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24a259c640 10 monclient(hunting): get_auth_request con 0x7f24ac0a49b0 auth_method 0 2024-04-19T01:48:46.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:46.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24a259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:46.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24a259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:46.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24aad9d640 1 -- 172.21.15.62:0/2987512634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24ac0a45d0 msgr2=0x7f24ac13eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-19T01:48:46.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24aad9d640 1 -- 172.21.15.62:0/2987512634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24ac0a45d0 msgr2=0x7f24ac13eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:48:46.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24aad9d640 1 --2- 172.21.15.62:0/2987512634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24ac0a45d0 0x7f24ac13eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:48:46.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24a259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:46.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24a259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:46.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24a259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:46.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.918+0000 7f24aad9d640 1 --2- 172.21.15.62:0/2987512634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24ac0a45d0 0x7f24ac13eb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:48:46.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24ab59e640 1 --2- 172.21.15.62:0/2987512634 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f24ac137390 0x7f24ac1cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:46.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24ab59e640 10 monclient(hunting): get_auth_request con 0x7f24ac137390 auth_method 0 2024-04-19T01:48:46.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24ab59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:46.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24ab59e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:46.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24ab59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:46.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24a259c640 10 monclient(hunting): handle_auth_done global_id 15255 payload 824 2024-04-19T01:48:46.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24a259c640 1 -- 172.21.15.62:0/2987512634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24ac0a45d0 msgr2=0x7f24ac13eb50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:48:46.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24a259c640 1 --2- 172.21.15.62:0/2987512634 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f24ac0a45d0 0x7f24ac13eb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:46.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24a259c640 1 -- 172.21.15.62:0/2987512634 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f24ac137390 msgr2=0x7f24ac1cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:46.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24a259c640 1 --2- 172.21.15.62:0/2987512634 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f24ac137390 0x7f24ac1cf7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:46.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24a259c640 10 monclient: _finish_hunting 0 2024-04-19T01:48:46.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24a259c640 1 monclient: found mon.noname-a 2024-04-19T01:48:46.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24a259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:46.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24a259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:48:46.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24a259c640 1 -- 172.21.15.62:0/2987512634 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24ac13d870 con 0x7f24ac0a49b0 2024-04-19T01:48:46.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24a259c640 1 -- 172.21.15.62:0/2987512634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ac0a49b0 msgr2=0x7f24ac1363d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:46.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24a259c640 1 --2- 172.21.15.62:0/2987512634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ac0a49b0 0x7f24ac1363d0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f24940ad000 tx=0x7f2494019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:46.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24a3fff640 1 -- 172.21.15.62:0/2987512634 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24940b9020 con 0x7f24ac0a49b0 2024-04-19T01:48:46.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:46.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24a3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:48:46.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24a3fff640 10 monclient: dump: 2024-04-19T01:48:46.934 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:46.934 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:46.934 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:46.934 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:46.934 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:46.934 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:46.934 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:46.934 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:46.934 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:46.934 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:46.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24a3fff640 10 monclient: _finish_auth 0 2024-04-19T01:48:46.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24a3fff640 10 monclient: _check_auth_tickets 2024-04-19T01:48:46.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:46.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24b115c640 5 monclient: authenticate success, global_id 15255 2024-04-19T01:48:46.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24a3fff640 1 -- 172.21.15.62:0/2987512634 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24940b56a0 con 0x7f24ac0a49b0 2024-04-19T01:48:46.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24b115c640 10 monclient: _renew_subs 2024-04-19T01:48:46.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24b115c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:46.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24b115c640 1 -- 172.21.15.62:0/2987512634 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24ac13dff0 con 0x7f24ac0a49b0 2024-04-19T01:48:46.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24a3fff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:46.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24a3fff640 1 -- 172.21.15.62:0/2987512634 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24940b0040 con 0x7f24ac0a49b0 2024-04-19T01:48:46.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:46.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:48:46.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24a3fff640 10 monclient: dump: 2024-04-19T01:48:46.935 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:46.936 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:46.936 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:46.936 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:46.936 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:46.936 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:46.936 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:46.936 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:46.936 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:46.936 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:46.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24a9d9b640 4 set_mon_vals no callback set 2024-04-19T01:48:46.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24b115c640 10 monclient: _renew_subs 2024-04-19T01:48:46.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24b115c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:46.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24b115c640 1 -- 172.21.15.62:0/2987512634 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24ac1204c0 con 0x7f24ac0a49b0 2024-04-19T01:48:46.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24b115c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:46.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24b115c640 1 --2- 172.21.15.62:0/2987512634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ac13c7f0 0x7f24ac1cfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:46.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24b115c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:46.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24b115c640 1 -- 172.21.15.62:0/2987512634 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f24ac13cbd0 con 0x7f24ac13c7f0 2024-04-19T01:48:46.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24aad9d640 1 --2- 172.21.15.62:0/2987512634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ac13c7f0 0x7f24ac1cfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:46.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24aad9d640 10 monclient: get_auth_request con 0x7f24ac13c7f0 auth_method 0 2024-04-19T01:48:46.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:46.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24aad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:46.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24aad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:46.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24a3fff640 1 -- 172.21.15.62:0/2987512634 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f24940bc020 con 0x7f24ac0a49b0 2024-04-19T01:48:46.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24aad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:46.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24aad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:46.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24aad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:46.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24a3fff640 1 --2- 172.21.15.62:0/2987512634 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f248c036470 0x7f248c038930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:46.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24a3fff640 1 -- 172.21.15.62:0/2987512634 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(325..325 src has 115..325) ==== 4346+0+0 (secure 0 0 0) 0x7f24940e2e10 con 0x7f24ac0a49b0 2024-04-19T01:48:46.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24aad9d640 10 monclient(hunting): handle_auth_done global_id 15258 payload 824 2024-04-19T01:48:46.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.922+0000 7f24aad9d640 1 -- 172.21.15.62:0/2987512634 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f24ac0692b0 con 0x7f24ac13c7f0 2024-04-19T01:48:46.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.926+0000 7f24ab59e640 1 --2- 172.21.15.62:0/2987512634 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f248c036470 0x7f248c038930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:46.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.926+0000 7f24ab59e640 10 monclient: get_auth_request con 0x7f248c036470 auth_method 0 2024-04-19T01:48:46.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.926+0000 7f24aad9d640 1 -- 172.21.15.62:0/2987512634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ac13c7f0 msgr2=0x7f24ac1cfd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:46.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.926+0000 7f24aad9d640 1 --2- 172.21.15.62:0/2987512634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ac13c7f0 0x7f24ac1cfd20 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f249c009fd0 tx=0x7f249c010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:46.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.926+0000 7f24a3fff640 1 -- 172.21.15.62:0/2987512634 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f249c01a030 con 0x7f24ac13c7f0 2024-04-19T01:48:46.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.926+0000 7f24a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:46.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.926+0000 7f24a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:48:46.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.926+0000 7f24a3fff640 10 monclient: dump: 2024-04-19T01:48:46.941 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:46.941 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:46.941 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:46.941 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:46.941 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:46.941 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:46.941 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:46.941 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:46.941 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:46.941 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:46.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.926+0000 7f24a3fff640 1 -- 172.21.15.62:0/2987512634 <== mon.0 v2:172.21.15.62:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f249c00ebe0 con 0x7f24ac13c7f0 2024-04-19T01:48:46.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.926+0000 7f24a3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:46.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.926+0000 7f24a3fff640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:48:46.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.926+0000 7f24a3fff640 1 -- 172.21.15.62:0/2987512634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ac13c7f0 msgr2=0x7f24ac1cfd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:46.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.926+0000 7f24a3fff640 1 --2- 172.21.15.62:0/2987512634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ac13c7f0 0x7f24ac1cfd20 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f249c009fd0 tx=0x7f249c010c20 comp rx=0 tx=0).stop 2024-04-19T01:48:46.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.926+0000 7f24a3fff640 1 -- 172.21.15.62:0/2987512634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ac13c7f0 msgr2=0x7f24ac1cfd20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:48:46.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.926+0000 7f24a3fff640 1 --2- 172.21.15.62:0/2987512634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ac13c7f0 0x7f24ac1cfd20 secure :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f249c009fd0 tx=0x7f249c010c20 comp rx=0 tx=0).stop 2024-04-19T01:48:46.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.926+0000 7f24ab59e640 1 --2- 172.21.15.62:0/2987512634 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f248c036470 0x7f248c038930 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f24a40054b0 tx=0x7f24a4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:46.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.934+0000 7f24b115c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:48:46.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.934+0000 7f24b115c640 1 --2- 172.21.15.62:0/2987512634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ac1d0260 0x7f24ac138480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:46.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.934+0000 7f24b115c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:46.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.934+0000 7f24b115c640 1 -- 172.21.15.62:0/2987512634 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f24ac1d0640 con 0x7f24ac1d0260 2024-04-19T01:48:46.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.934+0000 7f24aad9d640 1 --2- 172.21.15.62:0/2987512634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ac1d0260 0x7f24ac138480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:46.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.934+0000 7f24aad9d640 10 monclient: get_auth_request con 0x7f24ac1d0260 auth_method 0 2024-04-19T01:48:46.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.934+0000 7f24aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:46.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.934+0000 7f24aad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:46.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.934+0000 7f24aad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:46.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.934+0000 7f24aad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:46.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.934+0000 7f24aad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:46.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.934+0000 7f24aad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:46.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.934+0000 7f24aad9d640 10 monclient(hunting): handle_auth_done global_id 15261 payload 824 2024-04-19T01:48:46.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.934+0000 7f24aad9d640 1 -- 172.21.15.62:0/2987512634 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f24ac0a3370 con 0x7f24ac1d0260 2024-04-19T01:48:46.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.934+0000 7f24aad9d640 1 -- 172.21.15.62:0/2987512634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ac1d0260 msgr2=0x7f24ac138480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:46.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.934+0000 7f24aad9d640 1 --2- 172.21.15.62:0/2987512634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ac1d0260 0x7f24ac138480 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f249c009fd0 tx=0x7f249c0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:46.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.938+0000 7f24a3fff640 1 -- 172.21.15.62:0/2987512634 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f249c00f040 con 0x7f24ac1d0260 2024-04-19T01:48:46.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.938+0000 7f24a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:46.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.938+0000 7f24a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:48:46.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.938+0000 7f24a3fff640 10 monclient: dump: 2024-04-19T01:48:46.943 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:46.944 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:46.944 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:46.944 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:46.944 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:46.944 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:46.944 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:46.944 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:46.944 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:46.944 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:46.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.938+0000 7f24a3fff640 1 -- 172.21.15.62:0/2987512634 <== mon.0 v2:172.21.15.62:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4719 (secure 0 0 0) 0x7f249c000e50 con 0x7f24ac1d0260 2024-04-19T01:48:46.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.938+0000 7f24a3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:48:46.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.938+0000 7f24a3fff640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:48:46.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.938+0000 7f24a3fff640 1 -- 172.21.15.62:0/2987512634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ac1d0260 msgr2=0x7f24ac138480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:46.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.938+0000 7f24a3fff640 1 --2- 172.21.15.62:0/2987512634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ac1d0260 0x7f24ac138480 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f249c009fd0 tx=0x7f249c0046d0 comp rx=0 tx=0).stop 2024-04-19T01:48:46.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.938+0000 7f24a3fff640 1 -- 172.21.15.62:0/2987512634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ac1d0260 msgr2=0x7f24ac138480 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:48:46.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.938+0000 7f24a3fff640 1 --2- 172.21.15.62:0/2987512634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ac1d0260 0x7f24ac138480 secure :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f249c009fd0 tx=0x7f249c0046d0 comp rx=0 tx=0).stop 2024-04-19T01:48:46.945 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-19T01:48:46.945 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "a", 2024-04-19T01:48:46.945 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 0, 2024-04-19T01:48:46.945 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "leader", 2024-04-19T01:48:46.945 INFO:teuthology.orchestra.run.smithi062.stdout: "election_epoch": 40, 2024-04-19T01:48:46.945 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 270500, 2024-04-19T01:48:46.945 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum": [ 2024-04-19T01:48:46.945 INFO:teuthology.orchestra.run.smithi062.stdout: 0, 2024-04-19T01:48:46.946 INFO:teuthology.orchestra.run.smithi062.stdout: 1, 2024-04-19T01:48:46.946 INFO:teuthology.orchestra.run.smithi062.stdout: 2 2024-04-19T01:48:46.946 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:48:46.946 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_age": 0, 2024-04-19T01:48:46.946 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:48:46.946 INFO:teuthology.orchestra.run.smithi062.stdout: "required_con": "2450521708007866372", 2024-04-19T01:48:46.946 INFO:teuthology.orchestra.run.smithi062.stdout: "required_mon": [ 2024-04-19T01:48:46.946 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:48:46.946 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:48:46.946 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:48:46.946 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:48:46.946 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:48:46.946 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:48:46.946 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:48:46.947 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:48:46.947 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:48:46.947 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:48:46.947 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:48:46.947 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:48:46.947 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:48:46.947 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_mon": [ 2024-04-19T01:48:46.947 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:48:46.947 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:48:46.947 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:48:46.947 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:48:46.947 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:48:46.947 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:48:46.947 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:48:46.947 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:48:46.948 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:48:46.948 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:48:46.948 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:48:46.948 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:48:46.948 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:46.948 INFO:teuthology.orchestra.run.smithi062.stdout: "outside_quorum": [], 2024-04-19T01:48:46.948 INFO:teuthology.orchestra.run.smithi062.stdout: "extra_probe_peers": [], 2024-04-19T01:48:46.948 INFO:teuthology.orchestra.run.smithi062.stdout: "sync_provider": [], 2024-04-19T01:48:46.948 INFO:teuthology.orchestra.run.smithi062.stdout: "monmap": { 2024-04-19T01:48:46.948 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 1, 2024-04-19T01:48:46.948 INFO:teuthology.orchestra.run.smithi062.stdout: "fsid": "6466e4b6-2520-485f-95b3-9401915fbbad", 2024-04-19T01:48:46.948 INFO:teuthology.orchestra.run.smithi062.stdout: "modified": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:48:46.948 INFO:teuthology.orchestra.run.smithi062.stdout: "created": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:48:46.948 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release": 19, 2024-04-19T01:48:46.948 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release_name": "squid", 2024-04-19T01:48:46.949 INFO:teuthology.orchestra.run.smithi062.stdout: "election_strategy": 1, 2024-04-19T01:48:46.949 INFO:teuthology.orchestra.run.smithi062.stdout: "disallowed_leaders: ": "", 2024-04-19T01:48:46.949 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false, 2024-04-19T01:48:46.949 INFO:teuthology.orchestra.run.smithi062.stdout: "tiebreaker_mon": "", 2024-04-19T01:48:46.949 INFO:teuthology.orchestra.run.smithi062.stdout: "removed_ranks: ": "", 2024-04-19T01:48:46.949 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:48:46.949 INFO:teuthology.orchestra.run.smithi062.stdout: "persistent": [ 2024-04-19T01:48:46.949 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:48:46.949 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:48:46.949 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:48:46.949 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:48:46.949 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:48:46.949 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:48:46.949 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:48:46.949 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:48:46.949 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:48:46.949 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:48:46.950 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:48:46.950 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:48:46.950 INFO:teuthology.orchestra.run.smithi062.stdout: "optional": [] 2024-04-19T01:48:46.950 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:46.950 INFO:teuthology.orchestra.run.smithi062.stdout: "mons": [ 2024-04-19T01:48:46.950 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:46.950 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 0, 2024-04-19T01:48:46.950 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "a", 2024-04-19T01:48:46.950 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:48:46.950 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:48:46.950 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:46.950 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:48:46.950 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3300", 2024-04-19T01:48:46.951 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:48:46.951 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:46.951 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:46.951 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:48:46.951 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789", 2024-04-19T01:48:46.951 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:48:46.951 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:48:46.951 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:48:46.951 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:46.951 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789/0", 2024-04-19T01:48:46.951 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6789/0", 2024-04-19T01:48:46.951 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:48:46.951 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:48:46.951 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:48:46.952 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:46.952 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:46.952 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 1, 2024-04-19T01:48:46.952 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "b", 2024-04-19T01:48:46.952 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:48:46.952 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:48:46.952 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:46.952 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:48:46.952 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:3300", 2024-04-19T01:48:46.952 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:48:46.952 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:46.952 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:46.952 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:48:46.952 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789", 2024-04-19T01:48:46.952 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:48:46.952 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:48:46.952 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:48:46.953 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:46.953 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789/0", 2024-04-19T01:48:46.953 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.71:6789/0", 2024-04-19T01:48:46.953 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:48:46.953 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:48:46.953 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:48:46.953 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:46.953 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:46.953 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 2, 2024-04-19T01:48:46.953 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "c", 2024-04-19T01:48:46.953 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:48:46.953 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:48:46.953 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:46.953 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:48:46.953 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3301", 2024-04-19T01:48:46.954 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:48:46.954 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:46.954 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:46.954 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:48:46.954 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790", 2024-04-19T01:48:46.954 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:48:46.954 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:48:46.954 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:48:46.954 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:46.954 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790/0", 2024-04-19T01:48:46.954 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6790/0", 2024-04-19T01:48:46.954 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:48:46.954 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:48:46.954 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:48:46.954 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:48:46.955 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:48:46.955 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:46.955 INFO:teuthology.orchestra.run.smithi062.stdout: "feature_map": { 2024-04-19T01:48:46.955 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": [ 2024-04-19T01:48:46.955 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:46.955 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:48:46.955 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:48:46.955 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:48:46.955 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:48:46.955 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:48:46.955 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": [ 2024-04-19T01:48:46.955 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:46.955 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:48:46.955 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:48:46.956 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 6 2024-04-19T01:48:46.956 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:48:46.956 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:48:46.956 INFO:teuthology.orchestra.run.smithi062.stdout: "client": [ 2024-04-19T01:48:46.956 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:46.956 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:48:46.956 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:48:46.956 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 5 2024-04-19T01:48:46.956 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:48:46.956 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:48:46.956 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:46.956 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false 2024-04-19T01:48:46.956 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-19T01:48:46.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.938+0000 7f24b115c640 1 -- 172.21.15.62:0/2987512634 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f248c036470 msgr2=0x7f248c038930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:46.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.938+0000 7f24b115c640 1 --2- 172.21.15.62:0/2987512634 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f248c036470 0x7f248c038930 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f24a40054b0 tx=0x7f24a4009040 comp rx=0 tx=0).stop 2024-04-19T01:48:46.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.938+0000 7f24b115c640 10 monclient: shutdown 2024-04-19T01:48:46.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.938+0000 7f24b115c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:46.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.938+0000 7f24b115c640 1 -- 172.21.15.62:0/2987512634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ac0a49b0 msgr2=0x7f24ac1363d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:46.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.938+0000 7f24b115c640 1 --2- 172.21.15.62:0/2987512634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f24ac0a49b0 0x7f24ac1363d0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f24940ad000 tx=0x7f2494019190 comp rx=0 tx=0).stop 2024-04-19T01:48:46.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.938+0000 7f24ab59e640 1 -- 172.21.15.62:0/2987512634 reap_dead start 2024-04-19T01:48:46.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.938+0000 7f24ab59e640 1 -- 172.21.15.62:0/2987512634 reap_dead start 2024-04-19T01:48:46.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.938+0000 7f24b115c640 1 -- 172.21.15.62:0/2987512634 shutdown_connections 2024-04-19T01:48:46.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.938+0000 7f24b115c640 1 -- 172.21.15.62:0/2987512634 >> 172.21.15.62:0/2987512634 conn(0x7f24ac0aa1d0 msgr2=0x7f24ac09fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:46.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.938+0000 7f24b115c640 1 -- 172.21.15.62:0/2987512634 shutdown_connections 2024-04-19T01:48:46.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:46.938+0000 7f24b115c640 1 -- 172.21.15.62:0/2987512634 wait complete. 2024-04-19T01:48:46.959 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-19T01:48:46.959 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-19T01:48:46.959 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-19T01:48:47.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eac529640 10 monclient: get_monmap_and_config 2024-04-19T01:48:47.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eac529640 10 monclient: build_initial_monmap 2024-04-19T01:48:47.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eac529640 10 monclient: monmap: 2024-04-19T01:48:47.076 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:47.076 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:47.076 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:48:47.081790+0000 2024-04-19T01:48:47.076 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:48:47.081790+0000 2024-04-19T01:48:47.076 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:47.076 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:47.076 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:48:47.076 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:48:47.076 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:48:47.076 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:48:47.076 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:48:47.076 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:47.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eac529640 1 Processor -- start 2024-04-19T01:48:47.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eac529640 1 -- start start 2024-04-19T01:48:47.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eac529640 10 monclient: init 2024-04-19T01:48:47.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eac529640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:47.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eac529640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-04-19T01:48:47.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eac529640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1ea40a47e0 0x7f1ea40a4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:47.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eac529640 10 monclient(hunting): picked mon.noname-c con 0x7f1ea40a47e0 addr v2:172.21.15.71:3300/0 2024-04-19T01:48:47.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eac529640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea40a5100 0x7f1ea41350b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:47.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eac529640 10 monclient(hunting): picked mon.noname-b con 0x7f1ea40a5100 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:47.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eac529640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1ea41355f0 addr v1:172.21.15.62:6789/0 2024-04-19T01:48:47.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eac529640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ea409f7c0 con 0x7f1ea41355f0 2024-04-19T01:48:47.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eac529640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:47.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eac529640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f1ea409bf00 con 0x7f1ea40a47e0 2024-04-19T01:48:47.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eac529640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:47.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eac529640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f1ea409ee90 con 0x7f1ea40a5100 2024-04-19T01:48:47.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eac529640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:47.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eac529640 10 monclient(hunting): authenticate will time out at 1312.901354s 2024-04-19T01:48:47.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1ea9ac4640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea40a5100 0x7f1ea41350b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:47.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eaaac6640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f1ea41355f0 0x7f1ea413c5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:56676/0 (socket says 172.21.15.62:56676) 2024-04-19T01:48:47.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1ea9ac4640 10 monclient(hunting): get_auth_request con 0x7f1ea40a5100 auth_method 0 2024-04-19T01:48:47.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1ea9ac4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:47.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1ea9ac4640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:47.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eaaac6640 1 -- 172.21.15.62:0/3964273216 learned_addr learned my addr 172.21.15.62:0/3964273216 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:48:47.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1ea9ac4640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:47.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eaa2c5640 1 --2- 172.21.15.62:0/3964273216 >> v2:172.21.15.71:3300/0 conn(0x7f1ea40a47e0 0x7f1ea40a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:47.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eaa2c5640 10 monclient(hunting): get_auth_request con 0x7f1ea40a47e0 auth_method 0 2024-04-19T01:48:47.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eaa2c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:47.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eaa2c5640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:47.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eaa2c5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:47.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1ea9ac4640 1 -- 172.21.15.62:0/3964273216 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea40a5100 msgr2=0x7f1ea41350b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:48:47.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1ea9ac4640 1 -- 172.21.15.62:0/3964273216 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea40a5100 msgr2=0x7f1ea41350b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:48:47.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1ea9ac4640 1 --2- 172.21.15.62:0/3964273216 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea40a5100 0x7f1ea41350b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:48:47.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1ea9ac4640 1 --2- 172.21.15.62:0/3964273216 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea40a5100 0x7f1ea41350b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:48:47.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eaa2c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:47.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eaa2c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:47.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eaa2c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:47.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eaa2c5640 10 monclient(hunting): handle_auth_done global_id 24514 payload 274 2024-04-19T01:48:47.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eaa2c5640 1 -- 172.21.15.62:0/3964273216 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea40a5100 msgr2=0x7f1ea41350b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:48:47.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eaa2c5640 1 --2- 172.21.15.62:0/3964273216 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea40a5100 0x7f1ea41350b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:47.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eaa2c5640 1 -- 172.21.15.62:0/3964273216 >> v1:172.21.15.62:6789/0 conn(0x7f1ea41355f0 legacy=0x7f1ea413c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:47.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eaa2c5640 10 monclient: _finish_hunting 0 2024-04-19T01:48:47.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eaa2c5640 1 monclient: found mon.noname-c 2024-04-19T01:48:47.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eaa2c5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:47.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eaa2c5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.71:3300/0 2024-04-19T01:48:47.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.078+0000 7f1eaa2c5640 1 -- 172.21.15.62:0/3964273216 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ea409c2d0 con 0x7f1ea40a47e0 2024-04-19T01:48:47.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.082+0000 7f1eaa2c5640 1 -- 172.21.15.62:0/3964273216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea40a47e0 msgr2=0x7f1ea40a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:47.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.082+0000 7f1eaa2c5640 1 --2- 172.21.15.62:0/3964273216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea40a47e0 0x7f1ea40a4bc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1e94009d90 tx=0x7f1e9401c740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f49e9953de521b56 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:47.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.082+0000 7f1ea92c3640 1 -- 172.21.15.62:0/3964273216 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e940b2070 con 0x7f1ea40a47e0 2024-04-19T01:48:47.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.082+0000 7f1ea92c3640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:47.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.082+0000 7f1ea92c3640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:48:47.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.082+0000 7f1ea92c3640 10 monclient: dump: 2024-04-19T01:48:47.084 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:47.084 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:47.084 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:47.085 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:47.085 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:47.085 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:47.085 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:47.085 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:47.085 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:47.085 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:47.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.082+0000 7f1ea92c3640 10 monclient: _finish_auth 0 2024-04-19T01:48:47.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.082+0000 7f1ea92c3640 10 monclient: _check_auth_tickets 2024-04-19T01:48:47.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.082+0000 7f1ea92c3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:47.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.082+0000 7f1eac529640 5 monclient: authenticate success, global_id 24514 2024-04-19T01:48:47.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.082+0000 7f1eac529640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:48:47.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.082+0000 7f1ea92c3640 1 -- 172.21.15.62:0/3964273216 <== mon.1 v2:172.21.15.71:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e940a5780 con 0x7f1ea40a47e0 2024-04-19T01:48:47.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.082+0000 7f1ea92c3640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:47.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.082+0000 7f1ea92c3640 1 -- 172.21.15.62:0/3964273216 <== mon.1 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e940a5b50 con 0x7f1ea40a47e0 2024-04-19T01:48:47.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.082+0000 7f1ea92c3640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:47.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.082+0000 7f1ea92c3640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:48:47.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.082+0000 7f1ea92c3640 10 monclient: dump: 2024-04-19T01:48:47.086 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:47.086 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:47.086 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:47.086 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:47.086 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:47.086 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:47.086 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:47.086 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:47.086 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:47.086 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:47.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.082+0000 7f1eac529640 10 monclient: get_monmap_and_config success 2024-04-19T01:48:47.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.082+0000 7f1eac529640 4 set_mon_vals no callback set 2024-04-19T01:48:47.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.082+0000 7f1eac529640 10 monclient: shutdown 2024-04-19T01:48:47.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.082+0000 7f1eac529640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:47.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.082+0000 7f1eac529640 1 -- 172.21.15.62:0/3964273216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea40a47e0 msgr2=0x7f1ea40a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:47.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.082+0000 7f1eac529640 1 --2- 172.21.15.62:0/3964273216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea40a47e0 0x7f1ea40a4bc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1e94009d90 tx=0x7f1e9401c740 comp rx=0 tx=0).stop 2024-04-19T01:48:47.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.082+0000 7f1eac529640 1 -- 172.21.15.62:0/3964273216 shutdown_connections 2024-04-19T01:48:47.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.082+0000 7f1eac529640 1 --2- 172.21.15.62:0/3964273216 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea40a5100 0x7f1ea41350b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:47.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.082+0000 7f1eac529640 1 --2- 172.21.15.62:0/3964273216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea40a47e0 0x7f1ea40a4bc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:47.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.082+0000 7f1eac529640 1 -- 172.21.15.62:0/3964273216 >> 172.21.15.62:0/3964273216 conn(0x7f1ea40a23d0 msgr2=0x7f1ea40a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:47.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.082+0000 7f1eac529640 1 -- 172.21.15.62:0/3964273216 shutdown_connections 2024-04-19T01:48:47.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.082+0000 7f1eac529640 1 -- 172.21.15.62:0/3964273216 wait complete. 2024-04-19T01:48:47.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eac529640 10 monclient: build_initial_monmap 2024-04-19T01:48:47.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eac529640 10 monclient: monmap: 2024-04-19T01:48:47.087 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:47.087 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:48:47.088 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:48:47.088 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:48:47.088 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:47.088 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:47.088 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:48:47.088 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:48:47.088 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:48:47.088 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:47.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eac529640 1 Processor -- start 2024-04-19T01:48:47.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eac529640 1 -- start start 2024-04-19T01:48:47.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eac529640 10 monclient: init 2024-04-19T01:48:47.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eac529640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:47.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eac529640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-19T01:48:47.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eac529640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea40a47e0 0x7f1ea4139470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:47.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eac529640 10 monclient(hunting): picked mon.noname-c con 0x7f1ea40a47e0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:47.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eac529640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea40a5100 0x7f1ea413b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:47.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eac529640 10 monclient(hunting): picked mon.noname-b con 0x7f1ea40a5100 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:48:47.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eac529640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ea41355f0 0x7f1ea4131350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:47.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eac529640 10 monclient(hunting): picked mon.noname-a con 0x7f1ea41355f0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:48:47.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eac529640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:47.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eac529640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea409f1d0 con 0x7f1ea41355f0 2024-04-19T01:48:47.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eac529640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:47.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eac529640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea409f050 con 0x7f1ea40a5100 2024-04-19T01:48:47.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eac529640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:47.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eac529640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f1ea409c760 con 0x7f1ea40a47e0 2024-04-19T01:48:47.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eac529640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:47.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eac529640 10 monclient(hunting): authenticate will time out at 1312.909230s 2024-04-19T01:48:47.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eaa2c5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea40a47e0 0x7f1ea4139470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:47.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eaaac6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ea41355f0 0x7f1ea4131350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:47.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eaa2c5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea40a47e0 0x7f1ea4139470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:58028/0 (socket says 172.21.15.62:58028) 2024-04-19T01:48:47.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eaa2c5640 1 -- 172.21.15.62:0/1240293536 learned_addr learned my addr 172.21.15.62:0/1240293536 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:47.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eaa2c5640 10 monclient(hunting): get_auth_request con 0x7f1ea40a47e0 auth_method 0 2024-04-19T01:48:47.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eaa2c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:47.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eaa2c5640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:47.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eaa2c5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:47.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eaaac6640 10 monclient(hunting): get_auth_request con 0x7f1ea41355f0 auth_method 0 2024-04-19T01:48:47.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eaaac6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:47.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eaaac6640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:47.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eaaac6640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:47.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eaa2c5640 1 -- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea40a47e0 msgr2=0x7f1ea4139470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-19T01:48:47.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eaa2c5640 1 -- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea40a47e0 msgr2=0x7f1ea4139470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:48:47.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eaa2c5640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea40a47e0 0x7f1ea4139470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:48:47.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eaaac6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:47.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eaaac6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:47.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eaa2c5640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea40a47e0 0x7f1ea4139470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:48:47.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eaaac6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:47.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1ea9ac4640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea40a5100 0x7f1ea413b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:47.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1ea9ac4640 10 monclient(hunting): get_auth_request con 0x7f1ea40a5100 auth_method 0 2024-04-19T01:48:47.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1ea9ac4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:47.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1ea9ac4640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:47.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1ea9ac4640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:47.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eaaac6640 10 monclient(hunting): handle_auth_done global_id 15264 payload 824 2024-04-19T01:48:47.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eaaac6640 1 -- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea40a47e0 msgr2=0x7f1ea4139470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:48:47.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eaaac6640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea40a47e0 0x7f1ea4139470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:47.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eaaac6640 1 -- 172.21.15.62:0/1240293536 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea40a5100 msgr2=0x7f1ea413b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:47.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eaaac6640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea40a5100 0x7f1ea413b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:47.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eaaac6640 10 monclient: _finish_hunting 0 2024-04-19T01:48:47.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eaaac6640 1 monclient: found mon.noname-a 2024-04-19T01:48:47.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eaaac6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:47.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eaaac6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:48:47.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1eaaac6640 1 -- 172.21.15.62:0/1240293536 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ea413ac50 con 0x7f1ea41355f0 2024-04-19T01:48:47.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.086+0000 7f1ea9ac4640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea40a5100 0x7f1ea413b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T01:48:47.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1eaaac6640 1 -- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ea41355f0 msgr2=0x7f1ea4131350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:47.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1eaaac6640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ea41355f0 0x7f1ea4131350 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c00c410 tx=0x7f1e9c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:47.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1ea2ffd640 1 -- 172.21.15.62:0/1240293536 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c017070 con 0x7f1ea41355f0 2024-04-19T01:48:47.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1ea2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:47.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1ea2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:48:47.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1ea2ffd640 10 monclient: dump: 2024-04-19T01:48:47.097 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:47.097 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:47.097 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:47.097 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:47.097 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:47.097 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:47.098 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:47.098 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:47.098 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:47.098 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:47.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1ea2ffd640 10 monclient: _finish_auth 0 2024-04-19T01:48:47.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1ea2ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:48:47.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1ea2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:47.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1eac529640 5 monclient: authenticate success, global_id 15264 2024-04-19T01:48:47.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1eac529640 10 monclient: _renew_subs 2024-04-19T01:48:47.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1eac529640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:47.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1eac529640 1 -- 172.21.15.62:0/1240293536 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ea40726d0 con 0x7f1ea41355f0 2024-04-19T01:48:47.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1ea2ffd640 1 -- 172.21.15.62:0/1240293536 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e9c0070d0 con 0x7f1ea41355f0 2024-04-19T01:48:47.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1ea2ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:47.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1ea2ffd640 1 -- 172.21.15.62:0/1240293536 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c007500 con 0x7f1ea41355f0 2024-04-19T01:48:47.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1ea2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:47.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1ea2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:48:47.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1ea2ffd640 10 monclient: dump: 2024-04-19T01:48:47.099 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:47.099 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:47.099 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:47.099 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:47.099 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:47.099 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:47.099 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:47.099 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:47.099 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:47.099 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:47.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1ea8ac2640 4 set_mon_vals no callback set 2024-04-19T01:48:47.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1eac529640 10 monclient: _renew_subs 2024-04-19T01:48:47.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1eac529640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:47.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1eac529640 1 -- 172.21.15.62:0/1240293536 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ea413c350 con 0x7f1ea41355f0 2024-04-19T01:48:47.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1eac529640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:47.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1eac529640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea4131890 0x7f1ea41cd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:47.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1eac529640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:47.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1eac529640 1 -- 172.21.15.62:0/1240293536 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f1ea40692b0 con 0x7f1ea4131890 2024-04-19T01:48:47.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1eaa2c5640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea4131890 0x7f1ea41cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:47.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1eaa2c5640 10 monclient: get_auth_request con 0x7f1ea4131890 auth_method 0 2024-04-19T01:48:47.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1eaa2c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:47.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1eaa2c5640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:47.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1eaa2c5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:47.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1ea2ffd640 1 -- 172.21.15.62:0/1240293536 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f1e9c01a070 con 0x7f1ea41355f0 2024-04-19T01:48:47.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1eaa2c5640 1 -- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea4131890 msgr2=0x7f1ea41cd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-19T01:48:47.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1eaa2c5640 1 -- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea4131890 msgr2=0x7f1ea41cd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:48:47.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1eaa2c5640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea4131890 0x7f1ea41cd640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:48:47.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1eaa2c5640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea4131890 0x7f1ea41cd640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:48:47.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1ea2ffd640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f1e90036420 0x7f1e900388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:47.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1ea2ffd640 1 -- 172.21.15.62:0/1240293536 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(325..325 src has 115..325) ==== 4346+0+0 (secure 0 0 0) 0x7f1e9c048610 con 0x7f1ea41355f0 2024-04-19T01:48:47.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1ea9ac4640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f1e90036420 0x7f1e900388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:47.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.090+0000 7f1ea9ac4640 10 monclient: get_auth_request con 0x7f1e90036420 auth_method 0 2024-04-19T01:48:47.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.094+0000 7f1ea9ac4640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f1e90036420 0x7f1e900388e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1e98006aa0 tx=0x7f1e98006a30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:47.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.294+0000 7f1eaa2c5640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea4131890 0x7f1ea41cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:47.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.294+0000 7f1eaa2c5640 10 monclient: get_auth_request con 0x7f1ea4131890 auth_method 0 2024-04-19T01:48:47.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.294+0000 7f1eaa2c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:47.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.294+0000 7f1eaa2c5640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:47.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.294+0000 7f1eaa2c5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:48:47.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.294+0000 7f1eaa2c5640 1 -- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea4131890 msgr2=0x7f1ea41cd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-19T01:48:47.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.294+0000 7f1eaa2c5640 1 -- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea4131890 msgr2=0x7f1ea41cd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:48:47.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.294+0000 7f1eaa2c5640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea4131890 0x7f1ea41cd640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:48:47.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.294+0000 7f1eaa2c5640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea4131890 0x7f1ea41cd640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:48:47.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.698+0000 7f1eaa2c5640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea4131890 0x7f1ea41cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:47.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.698+0000 7f1eaa2c5640 10 monclient: get_auth_request con 0x7f1ea4131890 auth_method 0 2024-04-19T01:48:47.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.698+0000 7f1eaa2c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:47.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.698+0000 7f1eaa2c5640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:47.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.698+0000 7f1eaa2c5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:48:47.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.698+0000 7f1eaa2c5640 1 -- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea4131890 msgr2=0x7f1ea41cd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-19T01:48:47.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.698+0000 7f1eaa2c5640 1 -- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea4131890 msgr2=0x7f1ea41cd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:48:47.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.698+0000 7f1eaa2c5640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea4131890 0x7f1ea41cd640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:48:47.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:47.698+0000 7f1eaa2c5640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea4131890 0x7f1ea41cd640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:48:48.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:48.502+0000 7f1eaa2c5640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea4131890 0x7f1ea41cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:48.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:48.502+0000 7f1eaa2c5640 10 monclient: get_auth_request con 0x7f1ea4131890 auth_method 0 2024-04-19T01:48:48.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:48.502+0000 7f1eaa2c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:48.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:48.502+0000 7f1eaa2c5640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:48.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:48.502+0000 7f1eaa2c5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:48:48.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:48.502+0000 7f1eaa2c5640 1 -- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea4131890 msgr2=0x7f1ea41cd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-19T01:48:48.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:48.502+0000 7f1eaa2c5640 1 -- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea4131890 msgr2=0x7f1ea41cd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:48:48.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:48.502+0000 7f1eaa2c5640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea4131890 0x7f1ea41cd640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:48:48.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:48.502+0000 7f1eaa2c5640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea4131890 0x7f1ea41cd640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:48:50.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:50.090+0000 7f1ea1ffb640 10 monclient: tick 2024-04-19T01:48:50.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:50.090+0000 7f1ea1ffb640 10 monclient: _check_auth_tickets 2024-04-19T01:48:50.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:50.090+0000 7f1ea1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:50.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:50.090+0000 7f1ea1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:50.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:50.106+0000 7f1eaa2c5640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea4131890 0x7f1ea41cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:50.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:50.106+0000 7f1eaa2c5640 10 monclient: get_auth_request con 0x7f1ea4131890 auth_method 0 2024-04-19T01:48:50.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:50.106+0000 7f1eaa2c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:50.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:50.106+0000 7f1eaa2c5640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:50.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:50.106+0000 7f1eaa2c5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:48:50.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:50.106+0000 7f1eaa2c5640 1 -- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea4131890 msgr2=0x7f1ea41cd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-19T01:48:50.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:50.106+0000 7f1eaa2c5640 1 -- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea4131890 msgr2=0x7f1ea41cd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:48:50.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:50.106+0000 7f1eaa2c5640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea4131890 0x7f1ea41cd640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:48:50.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:50.106+0000 7f1eaa2c5640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea4131890 0x7f1ea41cd640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-19T01:48:51.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.090+0000 7f1ea1ffb640 10 monclient: tick 2024-04-19T01:48:51.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.090+0000 7f1ea1ffb640 10 monclient: _check_auth_tickets 2024-04-19T01:48:51.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.090+0000 7f1ea1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:51.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.090+0000 7f1ea1ffb640 5 monclient: _check_tell_commands timeout tell command 1 2024-04-19T01:48:51.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.090+0000 7f1ea1ffb640 1 -- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea4131890 msgr2=0x7f1ea41cd640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:48:51.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.090+0000 7f1ea1ffb640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea4131890 0x7f1ea41cd640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:51.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.090+0000 7f1ea1ffb640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1e88003550 0x7f1e880059e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:51.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.090+0000 7f1ea1ffb640 1 -- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea4131890 msgr2=0x7f1ea41cd640 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:48:51.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.090+0000 7f1ea1ffb640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea4131890 0x7f1ea41cd640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:51.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.090+0000 7f1ea1ffb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:51.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.090+0000 7f1ea1ffb640 1 -- 172.21.15.62:0/1240293536 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f1ea40692b0 con 0x7f1e88003550 2024-04-19T01:48:51.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.090+0000 7f1eaa2c5640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1e88003550 0x7f1e880059e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:51.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.090+0000 7f1eaa2c5640 10 monclient: get_auth_request con 0x7f1e88003550 auth_method 0 2024-04-19T01:48:51.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.090+0000 7f1eaa2c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:51.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.090+0000 7f1eaa2c5640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:51.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.090+0000 7f1eaa2c5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:51.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.090+0000 7f1eaa2c5640 1 -- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1e88003550 msgr2=0x7f1e880059e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-19T01:48:51.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.090+0000 7f1eaa2c5640 1 -- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1e88003550 msgr2=0x7f1e880059e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:48:51.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.090+0000 7f1eaa2c5640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1e88003550 0x7f1e880059e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:48:51.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.090+0000 7f1eaa2c5640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1e88003550 0x7f1e880059e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:48:51.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.293+0000 7f1eaa2c5640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1e88003550 0x7f1e880059e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:51.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.293+0000 7f1eaa2c5640 10 monclient: get_auth_request con 0x7f1e88003550 auth_method 0 2024-04-19T01:48:51.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.293+0000 7f1eaa2c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:51.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.293+0000 7f1eaa2c5640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:51.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.293+0000 7f1eaa2c5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:48:51.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.293+0000 7f1eaa2c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:51.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.293+0000 7f1eaa2c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:51.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.293+0000 7f1eaa2c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:51.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.293+0000 7f1eaa2c5640 10 monclient(hunting): handle_auth_done global_id 34100 payload 824 2024-04-19T01:48:51.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.293+0000 7f1eaa2c5640 1 -- 172.21.15.62:0/1240293536 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1e880060c0 con 0x7f1e88003550 2024-04-19T01:48:51.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.293+0000 7f1eaa2c5640 1 -- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1e88003550 msgr2=0x7f1e880059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:51.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.293+0000 7f1eaa2c5640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1e88003550 0x7f1e880059e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1e9401cfe0 tx=0x7f1e940194e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:51.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.297+0000 7f1ea2ffd640 1 -- 172.21.15.62:0/1240293536 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e940b2040 con 0x7f1e88003550 2024-04-19T01:48:51.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.297+0000 7f1ea2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:51.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.297+0000 7f1ea2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:48:51.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.297+0000 7f1ea2ffd640 10 monclient: dump: 2024-04-19T01:48:51.292 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:51.292 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:51.292 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:51.292 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:51.292 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:51.292 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:51.292 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:51.292 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:51.293 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:51.293 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:51.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.297+0000 7f1ea2ffd640 1 -- 172.21.15.62:0/1240293536 <== mon.2 v2:172.21.15.62:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1e940c1480 con 0x7f1e88003550 2024-04-19T01:48:51.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.297+0000 7f1ea2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:51.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.297+0000 7f1ea2ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:48:51.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.297+0000 7f1ea2ffd640 1 -- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1e88003550 msgr2=0x7f1e880059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:51.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.297+0000 7f1ea2ffd640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1e88003550 0x7f1e880059e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1e9401cfe0 tx=0x7f1e940194e0 comp rx=0 tx=0).stop 2024-04-19T01:48:51.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.297+0000 7f1ea2ffd640 1 -- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1e88003550 msgr2=0x7f1e880059e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:48:51.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.297+0000 7f1ea2ffd640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1e88003550 0x7f1e880059e0 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1e9401cfe0 tx=0x7f1e940194e0 comp rx=0 tx=0).stop 2024-04-19T01:48:51.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.305+0000 7f1eac529640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:48:51.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.305+0000 7f1eac529640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea41cdb80 0x7f1ea41cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:51.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.305+0000 7f1eac529640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:51.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.305+0000 7f1eac529640 1 -- 172.21.15.62:0/1240293536 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f1ea41ce4d0 con 0x7f1ea41cdb80 2024-04-19T01:48:51.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.305+0000 7f1eaa2c5640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea41cdb80 0x7f1ea41cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:51.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.305+0000 7f1eaa2c5640 10 monclient: get_auth_request con 0x7f1ea41cdb80 auth_method 0 2024-04-19T01:48:51.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.305+0000 7f1eaa2c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:51.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.305+0000 7f1eaa2c5640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:51.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.305+0000 7f1eaa2c5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:51.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.305+0000 7f1eaa2c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:51.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.305+0000 7f1eaa2c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:51.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.305+0000 7f1eaa2c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:51.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.305+0000 7f1eaa2c5640 10 monclient(hunting): handle_auth_done global_id 34103 payload 824 2024-04-19T01:48:51.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.305+0000 7f1eaa2c5640 1 -- 172.21.15.62:0/1240293536 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1ea406b980 con 0x7f1ea41cdb80 2024-04-19T01:48:51.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.309+0000 7f1eaa2c5640 1 -- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea41cdb80 msgr2=0x7f1ea41cdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:51.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.309+0000 7f1eaa2c5640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea41cdb80 0x7f1ea41cdf90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1e9401c780 tx=0x7f1e94009750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:51.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.309+0000 7f1ea2ffd640 1 -- 172.21.15.62:0/1240293536 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e940b2050 con 0x7f1ea41cdb80 2024-04-19T01:48:51.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.309+0000 7f1ea2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:51.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.309+0000 7f1ea2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:48:51.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.309+0000 7f1ea2ffd640 10 monclient: dump: 2024-04-19T01:48:51.305 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:51.305 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:51.305 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:51.305 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:51.305 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:51.305 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:51.305 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:51.305 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:51.305 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:51.306 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:51.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.309+0000 7f1ea2ffd640 1 -- 172.21.15.62:0/1240293536 <== mon.2 v2:172.21.15.62:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7f1e940d1070 con 0x7f1ea41cdb80 2024-04-19T01:48:51.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.309+0000 7f1ea2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:48:51.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.309+0000 7f1ea2ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:48:51.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.309+0000 7f1ea2ffd640 1 -- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea41cdb80 msgr2=0x7f1ea41cdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:51.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.309+0000 7f1ea2ffd640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea41cdb80 0x7f1ea41cdf90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1e9401c780 tx=0x7f1e94009750 comp rx=0 tx=0).stop 2024-04-19T01:48:51.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.309+0000 7f1ea2ffd640 1 -- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea41cdb80 msgr2=0x7f1ea41cdf90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:48:51.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.309+0000 7f1ea2ffd640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1ea41cdb80 0x7f1ea41cdf90 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1e9401c780 tx=0x7f1e94009750 comp rx=0 tx=0).stop 2024-04-19T01:48:51.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.309+0000 7f1eaaac6640 1 -- 172.21.15.62:0/1240293536 reap_dead start 2024-04-19T01:48:51.307 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-19T01:48:51.308 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "c", 2024-04-19T01:48:51.308 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 2, 2024-04-19T01:48:51.308 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "peon", 2024-04-19T01:48:51.308 INFO:teuthology.orchestra.run.smithi062.stdout: "election_epoch": 40, 2024-04-19T01:48:51.308 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 15051, 2024-04-19T01:48:51.308 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum": [ 2024-04-19T01:48:51.308 INFO:teuthology.orchestra.run.smithi062.stdout: 0, 2024-04-19T01:48:51.308 INFO:teuthology.orchestra.run.smithi062.stdout: 1, 2024-04-19T01:48:51.308 INFO:teuthology.orchestra.run.smithi062.stdout: 2 2024-04-19T01:48:51.308 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:48:51.308 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_age": 4, 2024-04-19T01:48:51.308 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:48:51.308 INFO:teuthology.orchestra.run.smithi062.stdout: "required_con": "2450521708007866372", 2024-04-19T01:48:51.309 INFO:teuthology.orchestra.run.smithi062.stdout: "required_mon": [ 2024-04-19T01:48:51.309 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:48:51.309 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:48:51.309 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:48:51.309 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:48:51.309 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:48:51.309 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:48:51.309 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:48:51.309 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:48:51.309 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:48:51.309 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:48:51.309 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:48:51.309 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:48:51.309 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:48:51.309 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_mon": [ 2024-04-19T01:48:51.310 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:48:51.310 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:48:51.310 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:48:51.310 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:48:51.310 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:48:51.310 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:48:51.310 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:48:51.310 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:48:51.310 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:48:51.310 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:48:51.310 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:48:51.310 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:48:51.310 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:51.310 INFO:teuthology.orchestra.run.smithi062.stdout: "outside_quorum": [], 2024-04-19T01:48:51.310 INFO:teuthology.orchestra.run.smithi062.stdout: "extra_probe_peers": [], 2024-04-19T01:48:51.311 INFO:teuthology.orchestra.run.smithi062.stdout: "sync_provider": [], 2024-04-19T01:48:51.311 INFO:teuthology.orchestra.run.smithi062.stdout: "monmap": { 2024-04-19T01:48:51.311 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 1, 2024-04-19T01:48:51.311 INFO:teuthology.orchestra.run.smithi062.stdout: "fsid": "6466e4b6-2520-485f-95b3-9401915fbbad", 2024-04-19T01:48:51.311 INFO:teuthology.orchestra.run.smithi062.stdout: "modified": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:48:51.311 INFO:teuthology.orchestra.run.smithi062.stdout: "created": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:48:51.311 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release": 19, 2024-04-19T01:48:51.311 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release_name": "squid", 2024-04-19T01:48:51.311 INFO:teuthology.orchestra.run.smithi062.stdout: "election_strategy": 1, 2024-04-19T01:48:51.311 INFO:teuthology.orchestra.run.smithi062.stdout: "disallowed_leaders: ": "", 2024-04-19T01:48:51.311 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false, 2024-04-19T01:48:51.311 INFO:teuthology.orchestra.run.smithi062.stdout: "tiebreaker_mon": "", 2024-04-19T01:48:51.311 INFO:teuthology.orchestra.run.smithi062.stdout: "removed_ranks: ": "", 2024-04-19T01:48:51.311 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:48:51.311 INFO:teuthology.orchestra.run.smithi062.stdout: "persistent": [ 2024-04-19T01:48:51.311 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:48:51.312 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:48:51.312 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:48:51.312 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:48:51.312 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:48:51.312 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:48:51.312 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:48:51.312 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:48:51.312 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:48:51.312 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:48:51.312 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:48:51.312 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:48:51.312 INFO:teuthology.orchestra.run.smithi062.stdout: "optional": [] 2024-04-19T01:48:51.312 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:51.312 INFO:teuthology.orchestra.run.smithi062.stdout: "mons": [ 2024-04-19T01:48:51.312 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:51.312 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 0, 2024-04-19T01:48:51.313 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "a", 2024-04-19T01:48:51.313 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:48:51.313 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:48:51.313 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:51.313 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:48:51.313 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3300", 2024-04-19T01:48:51.313 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:48:51.313 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:51.313 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:51.313 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:48:51.313 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789", 2024-04-19T01:48:51.313 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:48:51.313 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:48:51.313 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:48:51.313 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:51.313 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789/0", 2024-04-19T01:48:51.313 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6789/0", 2024-04-19T01:48:51.314 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:48:51.314 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:48:51.314 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:48:51.314 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:51.314 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:51.314 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 1, 2024-04-19T01:48:51.314 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "b", 2024-04-19T01:48:51.314 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:48:51.314 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:48:51.314 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:51.314 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:48:51.314 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:3300", 2024-04-19T01:48:51.314 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:48:51.314 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:51.314 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:51.314 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:48:51.315 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789", 2024-04-19T01:48:51.315 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:48:51.315 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:48:51.315 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:48:51.315 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:51.315 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789/0", 2024-04-19T01:48:51.315 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.71:6789/0", 2024-04-19T01:48:51.315 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:48:51.315 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:48:51.315 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:48:51.315 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:51.315 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:51.315 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 2, 2024-04-19T01:48:51.316 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "c", 2024-04-19T01:48:51.316 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:48:51.316 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:48:51.316 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:51.316 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:48:51.316 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3301", 2024-04-19T01:48:51.316 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:48:51.316 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:51.316 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:51.316 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:48:51.316 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790", 2024-04-19T01:48:51.316 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:48:51.316 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:48:51.316 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:48:51.316 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:51.316 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790/0", 2024-04-19T01:48:51.317 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6790/0", 2024-04-19T01:48:51.317 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:48:51.317 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:48:51.317 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:48:51.317 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:48:51.317 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:48:51.317 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:51.317 INFO:teuthology.orchestra.run.smithi062.stdout: "feature_map": { 2024-04-19T01:48:51.317 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": [ 2024-04-19T01:48:51.317 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:51.317 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:48:51.317 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:48:51.318 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:48:51.318 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:48:51.318 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:48:51.318 INFO:teuthology.orchestra.run.smithi062.stdout: "client": [ 2024-04-19T01:48:51.318 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:51.318 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:48:51.318 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:48:51.318 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:48:51.318 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:48:51.318 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:48:51.318 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:51.318 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false 2024-04-19T01:48:51.318 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-19T01:48:51.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.309+0000 7f1eac529640 1 -- 172.21.15.62:0/1240293536 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f1e90036420 msgr2=0x7f1e900388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:51.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.309+0000 7f1eac529640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f1e90036420 0x7f1e900388e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1e98006aa0 tx=0x7f1e98006a30 comp rx=0 tx=0).stop 2024-04-19T01:48:51.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.309+0000 7f1eac529640 10 monclient: shutdown 2024-04-19T01:48:51.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.309+0000 7f1eac529640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:51.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.309+0000 7f1eac529640 1 -- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ea41355f0 msgr2=0x7f1ea4131350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:51.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.309+0000 7f1eac529640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ea41355f0 0x7f1ea4131350 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c00c410 tx=0x7f1e9c00c8d0 comp rx=0 tx=0).stop 2024-04-19T01:48:51.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.309+0000 7f1eac529640 1 -- 172.21.15.62:0/1240293536 shutdown_connections 2024-04-19T01:48:51.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.309+0000 7f1eac529640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ea41355f0 0x7f1ea4131350 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:51.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.309+0000 7f1eac529640 1 --2- 172.21.15.62:0/1240293536 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f1e90036420 0x7f1e900388e0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:51.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.309+0000 7f1eac529640 1 -- 172.21.15.62:0/1240293536 >> 172.21.15.62:0/1240293536 conn(0x7f1ea40a23d0 msgr2=0x7f1ea40a11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:51.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.309+0000 7f1eac529640 1 -- 172.21.15.62:0/1240293536 shutdown_connections 2024-04-19T01:48:51.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.309+0000 7f1eac529640 1 -- 172.21.15.62:0/1240293536 wait complete. 2024-04-19T01:48:51.331 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-19T01:48:51.331 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-19T01:48:51.331 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T01:48:51.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.465+0000 7f5de7c36640 10 monclient: get_monmap_and_config 2024-04-19T01:48:51.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.465+0000 7f5de7c36640 10 monclient: build_initial_monmap 2024-04-19T01:48:51.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.465+0000 7f5de7c36640 10 monclient: monmap: 2024-04-19T01:48:51.463 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:51.463 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:51.464 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:48:51.469602+0000 2024-04-19T01:48:51.464 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:48:51.469602+0000 2024-04-19T01:48:51.464 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:51.464 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:51.464 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:48:51.464 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:48:51.464 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:48:51.464 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:48:51.464 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:48:51.464 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:51.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de7c36640 1 Processor -- start 2024-04-19T01:48:51.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de7c36640 1 -- start start 2024-04-19T01:48:51.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de7c36640 10 monclient: init 2024-04-19T01:48:51.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de7c36640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:51.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de7c36640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-04-19T01:48:51.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de7c36640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5de00a3410 0x7f5de00a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:51.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de7c36640 10 monclient(hunting): picked mon.noname-a con 0x7f5de00a3410 addr v2:172.21.15.62:3300/0 2024-04-19T01:48:51.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de7c36640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5de00a2900 addr v1:172.21.15.62:6789/0 2024-04-19T01:48:51.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de7c36640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5de00a2d50 0x7f5de00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:51.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de7c36640 10 monclient(hunting): picked mon.noname-b con 0x7f5de00a2d50 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:51.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de7c36640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5de00a1430 con 0x7f5de00a2900 2024-04-19T01:48:51.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de7c36640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:51.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de7c36640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f5de00a1710 con 0x7f5de00a3410 2024-04-19T01:48:51.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de7c36640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:51.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de7c36640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f5de00a19f0 con 0x7f5de00a2d50 2024-04-19T01:48:51.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de7c36640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:51.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de7c36640 10 monclient(hunting): authenticate will time out at 1317.289064s 2024-04-19T01:48:51.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de61d3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5de00a2d50 0x7f5de00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:51.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de59d2640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5de00a3410 0x7f5de00a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:51.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de61d3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5de00a2d50 0x7f5de00a4de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:58130/0 (socket says 172.21.15.62:58130) 2024-04-19T01:48:51.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de61d3640 1 -- 172.21.15.62:0/1559493475 learned_addr learned my addr 172.21.15.62:0/1559493475 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:51.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de61d3640 10 monclient(hunting): get_auth_request con 0x7f5de00a2d50 auth_method 0 2024-04-19T01:48:51.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de61d3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:51.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de61d3640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:51.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de61d3640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:51.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de59d2640 10 monclient(hunting): get_auth_request con 0x7f5de00a3410 auth_method 0 2024-04-19T01:48:51.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de59d2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:51.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de59d2640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:51.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de59d2640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:51.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de61d3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:51.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de61d3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:51.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de61d3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:51.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de59d2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:51.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de59d2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:51.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de59d2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:51.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de61d3640 10 monclient(hunting): handle_auth_done global_id 34106 payload 274 2024-04-19T01:48:51.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de61d3640 1 -- 172.21.15.62:0/1559493475 >> v2:172.21.15.62:3300/0 conn(0x7f5de00a3410 msgr2=0x7f5de00a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:51.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de61d3640 1 --2- 172.21.15.62:0/1559493475 >> v2:172.21.15.62:3300/0 conn(0x7f5de00a3410 0x7f5de00a37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:51.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de61d3640 1 -- 172.21.15.62:0/1559493475 >> v1:172.21.15.62:6789/0 conn(0x7f5de00a2900 legacy=0x7f5de00a46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:51.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de61d3640 10 monclient: _finish_hunting 0 2024-04-19T01:48:51.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de61d3640 1 monclient: found mon.noname-b 2024-04-19T01:48:51.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de61d3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:51.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de61d3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:48:51.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de61d3640 1 -- 172.21.15.62:0/1559493475 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5de00a3d30 con 0x7f5de00a2d50 2024-04-19T01:48:51.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.469+0000 7f5de59d2640 1 --2- 172.21.15.62:0/1559493475 >> v2:172.21.15.62:3300/0 conn(0x7f5de00a3410 0x7f5de00a37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:48:51.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de61d3640 1 -- 172.21.15.62:0/1559493475 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5de00a2d50 msgr2=0x7f5de00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:51.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de61d3640 1 --2- 172.21.15.62:0/1559493475 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5de00a2d50 0x7f5de00a4de0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5dd8009e50 tx=0x7f5dd801c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3594e37ead7479f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:51.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de49d0640 1 -- 172.21.15.62:0/1559493475 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd80a5640 con 0x7f5de00a2d50 2024-04-19T01:48:51.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de49d0640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:51.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de49d0640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:48:51.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de49d0640 10 monclient: dump: 2024-04-19T01:48:51.473 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:51.473 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:51.473 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:51.473 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:51.473 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:51.474 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:51.474 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:51.474 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:51.474 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:51.474 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:51.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de49d0640 10 monclient: _finish_auth 0 2024-04-19T01:48:51.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de49d0640 10 monclient: _check_auth_tickets 2024-04-19T01:48:51.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de49d0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:51.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de49d0640 1 -- 172.21.15.62:0/1559493475 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dd80a57e0 con 0x7f5de00a2d50 2024-04-19T01:48:51.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de7c36640 5 monclient: authenticate success, global_id 34106 2024-04-19T01:48:51.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de7c36640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:48:51.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de49d0640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:51.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de7c36640 10 monclient: get_monmap_and_config success 2024-04-19T01:48:51.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de7c36640 4 set_mon_vals no callback set 2024-04-19T01:48:51.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de49d0640 1 -- 172.21.15.62:0/1559493475 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd80a5bb0 con 0x7f5de00a2d50 2024-04-19T01:48:51.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de49d0640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:51.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de49d0640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:48:51.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de49d0640 10 monclient: dump: 2024-04-19T01:48:51.475 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:51.475 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:51.475 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:51.475 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:51.475 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:51.475 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:51.475 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:51.475 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:51.475 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:51.475 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:51.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de7c36640 10 monclient: shutdown 2024-04-19T01:48:51.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de7c36640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:51.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de7c36640 1 -- 172.21.15.62:0/1559493475 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5de00a2d50 msgr2=0x7f5de00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:51.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de7c36640 1 --2- 172.21.15.62:0/1559493475 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5de00a2d50 0x7f5de00a4de0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5dd8009e50 tx=0x7f5dd801c8f0 comp rx=0 tx=0).stop 2024-04-19T01:48:51.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de7c36640 1 -- 172.21.15.62:0/1559493475 shutdown_connections 2024-04-19T01:48:51.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de7c36640 1 --2- 172.21.15.62:0/1559493475 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5de00a2d50 0x7f5de00a4de0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:51.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de7c36640 1 --2- 172.21.15.62:0/1559493475 >> v2:172.21.15.62:3300/0 conn(0x7f5de00a3410 0x7f5de00a37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:51.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de7c36640 1 -- 172.21.15.62:0/1559493475 >> 172.21.15.62:0/1559493475 conn(0x7f5de00aa2d0 msgr2=0x7f5de00a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:51.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de7c36640 1 -- 172.21.15.62:0/1559493475 shutdown_connections 2024-04-19T01:48:51.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de7c36640 1 -- 172.21.15.62:0/1559493475 wait complete. 2024-04-19T01:48:51.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de7c36640 10 monclient: build_initial_monmap 2024-04-19T01:48:51.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de7c36640 10 monclient: monmap: 2024-04-19T01:48:51.476 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:51.476 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:48:51.476 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:48:51.477 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:48:51.477 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:51.477 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:51.477 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:48:51.477 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:48:51.477 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:48:51.477 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:51.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de7c36640 1 Processor -- start 2024-04-19T01:48:51.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de7c36640 1 -- start start 2024-04-19T01:48:51.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.473+0000 7f5de7c36640 10 monclient: init 2024-04-19T01:48:51.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de7c36640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:51.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de7c36640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-19T01:48:51.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de7c36640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5de00a2900 0x7f5de013e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:51.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de7c36640 10 monclient(hunting): picked mon.noname-a con 0x7f5de00a2900 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:48:51.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de7c36640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5de00a2d50 0x7f5de0124fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:51.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de7c36640 10 monclient(hunting): picked mon.noname-b con 0x7f5de00a2d50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:48:51.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de7c36640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5de00a3410 0x7f5de0125500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:51.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de7c36640 10 monclient(hunting): picked mon.noname-c con 0x7f5de00a3410 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:51.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de7c36640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:51.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de7c36640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5de0142380 con 0x7f5de00a2900 2024-04-19T01:48:51.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de7c36640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:51.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de7c36640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5de0142200 con 0x7f5de00a2d50 2024-04-19T01:48:51.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de7c36640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:51.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de7c36640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f5de0142500 con 0x7f5de00a3410 2024-04-19T01:48:51.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de7c36640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:51.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de7c36640 10 monclient(hunting): authenticate will time out at 1317.296414s 2024-04-19T01:48:51.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de59d2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5de00a2900 0x7f5de013e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:51.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de61d3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5de00a3410 0x7f5de0125500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:51.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de59d2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5de00a2900 0x7f5de013e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47742/0 (socket says 172.21.15.62:47742) 2024-04-19T01:48:51.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de59d2640 1 -- 172.21.15.62:0/2897362909 learned_addr learned my addr 172.21.15.62:0/2897362909 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:51.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de59d2640 10 monclient(hunting): get_auth_request con 0x7f5de00a2900 auth_method 0 2024-04-19T01:48:51.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de59d2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:51.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de59d2640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:51.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de59d2640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:51.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de61d3640 10 monclient(hunting): get_auth_request con 0x7f5de00a3410 auth_method 0 2024-04-19T01:48:51.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de61d3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:51.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de61d3640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:51.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de61d3640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:51.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de61d3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:51.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de61d3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:51.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de61d3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:51.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de59d2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:51.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de59d2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:51.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de59d2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:51.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de51d1640 1 --2- 172.21.15.62:0/2897362909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5de00a2d50 0x7f5de0124fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:51.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de61d3640 10 monclient(hunting): handle_auth_done global_id 34109 payload 824 2024-04-19T01:48:51.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de61d3640 1 -- 172.21.15.62:0/2897362909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5de00a2d50 msgr2=0x7f5de0124fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:51.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de61d3640 1 --2- 172.21.15.62:0/2897362909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5de00a2d50 0x7f5de0124fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:51.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de61d3640 1 -- 172.21.15.62:0/2897362909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5de00a2900 msgr2=0x7f5de013e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:51.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de61d3640 1 --2- 172.21.15.62:0/2897362909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5de00a2900 0x7f5de013e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:51.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de61d3640 10 monclient: _finish_hunting 0 2024-04-19T01:48:51.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de61d3640 1 monclient: found mon.noname-c 2024-04-19T01:48:51.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de61d3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:51.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de61d3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:48:51.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de61d3640 1 -- 172.21.15.62:0/2897362909 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5de0128f00 con 0x7f5de00a3410 2024-04-19T01:48:51.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de51d1640 10 monclient: get_auth_request con 0x7f5de00a2d50 auth_method 0 2024-04-19T01:48:51.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de59d2640 1 --2- 172.21.15.62:0/2897362909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5de00a2900 0x7f5de013e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:48:51.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de51d1640 1 --2- 172.21.15.62:0/2897362909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5de00a2d50 0x7f5de0124fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:48:51.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de61d3640 1 -- 172.21.15.62:0/2897362909 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5de00a3410 msgr2=0x7f5de0125500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:51.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de61d3640 1 --2- 172.21.15.62:0/2897362909 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5de00a3410 0x7f5de0125500 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5dd80ae000 tx=0x7f5dd8019690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:51.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5dd67fc640 1 -- 172.21.15.62:0/2897362909 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd80bc020 con 0x7f5de00a3410 2024-04-19T01:48:51.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5dd67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:51.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5dd67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:48:51.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5dd67fc640 10 monclient: dump: 2024-04-19T01:48:51.484 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:51.484 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:51.484 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:51.484 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:51.484 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:51.484 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:51.484 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:51.485 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:51.485 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:51.485 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:51.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5dd67fc640 10 monclient: _finish_auth 0 2024-04-19T01:48:51.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5dd67fc640 10 monclient: _check_auth_tickets 2024-04-19T01:48:51.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5dd67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:51.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5dd67fc640 1 -- 172.21.15.62:0/2897362909 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dd80b8a90 con 0x7f5de00a3410 2024-04-19T01:48:51.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5dd67fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:51.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5dd67fc640 1 -- 172.21.15.62:0/2897362909 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd80a5640 con 0x7f5de00a3410 2024-04-19T01:48:51.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5dd67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:51.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5dd67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:48:51.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5dd67fc640 10 monclient: dump: 2024-04-19T01:48:51.485 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:51.485 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:51.485 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:51.486 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:51.486 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:51.486 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:51.486 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:51.486 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:51.486 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:51.486 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:51.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de7c36640 5 monclient: authenticate success, global_id 34109 2024-04-19T01:48:51.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5dd7fff640 4 set_mon_vals no callback set 2024-04-19T01:48:51.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de7c36640 10 monclient: _renew_subs 2024-04-19T01:48:51.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de7c36640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:48:51.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de7c36640 1 -- 172.21.15.62:0/2897362909 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5de00a0040 con 0x7f5de00a3410 2024-04-19T01:48:51.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de7c36640 10 monclient: _renew_subs 2024-04-19T01:48:51.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de7c36640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:48:51.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.477+0000 7f5de7c36640 1 -- 172.21.15.62:0/2897362909 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5de0128c20 con 0x7f5de00a3410 2024-04-19T01:48:51.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.481+0000 7f5de7c36640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:51.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.481+0000 7f5de7c36640 1 --2- 172.21.15.62:0/2897362909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5de0125a40 0x7f5de0125e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:51.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.481+0000 7f5de7c36640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:51.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.481+0000 7f5de7c36640 1 -- 172.21.15.62:0/2897362909 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5de01275b0 con 0x7f5de0125a40 2024-04-19T01:48:51.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.481+0000 7f5dd67fc640 1 -- 172.21.15.62:0/2897362909 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f5dd80bf020 con 0x7f5de00a3410 2024-04-19T01:48:51.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.481+0000 7f5dd67fc640 1 --2- 172.21.15.62:0/2897362909 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5dc4036480 0x7f5dc4038940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:51.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.481+0000 7f5de59d2640 1 --2- 172.21.15.62:0/2897362909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5de0125a40 0x7f5de0125e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:51.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.481+0000 7f5de59d2640 10 monclient: get_auth_request con 0x7f5de0125a40 auth_method 0 2024-04-19T01:48:51.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.481+0000 7f5de59d2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:51.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.481+0000 7f5de59d2640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:51.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.481+0000 7f5de59d2640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:51.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.481+0000 7f5dd67fc640 1 -- 172.21.15.62:0/2897362909 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(325..325 src has 115..325) ==== 4346+0+0 (secure 0 0 0) 0x7f5dd80e4150 con 0x7f5de00a3410 2024-04-19T01:48:51.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.481+0000 7f5de59d2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:51.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.481+0000 7f5de59d2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:51.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.481+0000 7f5de59d2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:51.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.481+0000 7f5de51d1640 1 --2- 172.21.15.62:0/2897362909 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5dc4036480 0x7f5dc4038940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:51.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.481+0000 7f5de51d1640 10 monclient: get_auth_request con 0x7f5dc4036480 auth_method 0 2024-04-19T01:48:51.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.481+0000 7f5de59d2640 10 monclient(hunting): handle_auth_done global_id 24520 payload 824 2024-04-19T01:48:51.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.481+0000 7f5de59d2640 1 -- 172.21.15.62:0/2897362909 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5de00692b0 con 0x7f5de0125a40 2024-04-19T01:48:51.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.481+0000 7f5de59d2640 1 -- 172.21.15.62:0/2897362909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5de0125a40 msgr2=0x7f5de0125e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:51.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.481+0000 7f5de59d2640 1 --2- 172.21.15.62:0/2897362909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5de0125a40 0x7f5de0125e20 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc011db0 tx=0x7f5ddc00f610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:51.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.481+0000 7f5de51d1640 1 --2- 172.21.15.62:0/2897362909 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5dc4036480 0x7f5dc4038940 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f5de0126810 tx=0x7f5dd0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:51.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.485+0000 7f5dd67fc640 1 -- 172.21.15.62:0/2897362909 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ddc019070 con 0x7f5de0125a40 2024-04-19T01:48:51.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.485+0000 7f5dd67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:51.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.485+0000 7f5dd67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:48:51.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.485+0000 7f5dd67fc640 10 monclient: dump: 2024-04-19T01:48:51.488 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:51.488 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:51.488 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:51.488 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:51.489 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:51.489 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:51.489 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:51.489 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:51.489 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:51.489 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:51.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.485+0000 7f5dd67fc640 1 -- 172.21.15.62:0/2897362909 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5ddc00fe70 con 0x7f5de0125a40 2024-04-19T01:48:51.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.485+0000 7f5dd67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:51.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.485+0000 7f5dd67fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:48:51.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.485+0000 7f5dd67fc640 1 -- 172.21.15.62:0/2897362909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5de0125a40 msgr2=0x7f5de0125e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:51.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.485+0000 7f5dd67fc640 1 --2- 172.21.15.62:0/2897362909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5de0125a40 0x7f5de0125e20 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc011db0 tx=0x7f5ddc00f610 comp rx=0 tx=0).stop 2024-04-19T01:48:51.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.485+0000 7f5dd67fc640 1 -- 172.21.15.62:0/2897362909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5de0125a40 msgr2=0x7f5de0125e20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:48:51.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.485+0000 7f5dd67fc640 1 --2- 172.21.15.62:0/2897362909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5de0125a40 0x7f5de0125e20 secure :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc011db0 tx=0x7f5ddc00f610 comp rx=0 tx=0).stop 2024-04-19T01:48:51.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.489+0000 7f5de7c36640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:48:51.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.489+0000 7f5de7c36640 1 --2- 172.21.15.62:0/2897362909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5de0120320 0x7f5de0120700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:51.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.489+0000 7f5de7c36640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:51.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.489+0000 7f5de7c36640 1 -- 172.21.15.62:0/2897362909 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5de00690c0 con 0x7f5de0120320 2024-04-19T01:48:51.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.489+0000 7f5de59d2640 1 --2- 172.21.15.62:0/2897362909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5de0120320 0x7f5de0120700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:51.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.489+0000 7f5de59d2640 10 monclient: get_auth_request con 0x7f5de0120320 auth_method 0 2024-04-19T01:48:51.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.489+0000 7f5de59d2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:51.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.489+0000 7f5de59d2640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:51.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.489+0000 7f5de59d2640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:51.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.489+0000 7f5de59d2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:51.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.489+0000 7f5de59d2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:51.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.489+0000 7f5de59d2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:51.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.493+0000 7f5de59d2640 10 monclient(hunting): handle_auth_done global_id 24523 payload 824 2024-04-19T01:48:51.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.493+0000 7f5de59d2640 1 -- 172.21.15.62:0/2897362909 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5de0126360 con 0x7f5de0120320 2024-04-19T01:48:51.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.493+0000 7f5de59d2640 1 -- 172.21.15.62:0/2897362909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5de0120320 msgr2=0x7f5de0120700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:51.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.493+0000 7f5de59d2640 1 --2- 172.21.15.62:0/2897362909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5de0120320 0x7f5de0120700 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc01b000 tx=0x7f5ddc0046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:51.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.493+0000 7f5dd67fc640 1 -- 172.21.15.62:0/2897362909 <== mon.1 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ddc019040 con 0x7f5de0120320 2024-04-19T01:48:51.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.493+0000 7f5dd67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:51.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.493+0000 7f5dd67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:48:51.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.493+0000 7f5dd67fc640 10 monclient: dump: 2024-04-19T01:48:51.493 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:51.493 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:51.493 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:51.493 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:51.493 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:51.493 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:51.493 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:51.493 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:51.493 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:51.493 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:51.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.493+0000 7f5dd67fc640 1 -- 172.21.15.62:0/2897362909 <== mon.1 v2:172.21.15.71:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4717 (secure 0 0 0) 0x7f5ddc01e870 con 0x7f5de0120320 2024-04-19T01:48:51.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.493+0000 7f5dd67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:48:51.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.493+0000 7f5dd67fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:48:51.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.493+0000 7f5dd67fc640 1 -- 172.21.15.62:0/2897362909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5de0120320 msgr2=0x7f5de0120700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:51.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.493+0000 7f5dd67fc640 1 --2- 172.21.15.62:0/2897362909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5de0120320 0x7f5de0120700 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc01b000 tx=0x7f5ddc0046d0 comp rx=0 tx=0).stop 2024-04-19T01:48:51.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.493+0000 7f5dd67fc640 1 -- 172.21.15.62:0/2897362909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5de0120320 msgr2=0x7f5de0120700 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:48:51.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.493+0000 7f5dd67fc640 1 --2- 172.21.15.62:0/2897362909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5de0120320 0x7f5de0120700 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:51.494 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-19T01:48:51.494 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "b", 2024-04-19T01:48:51.494 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 1, 2024-04-19T01:48:51.494 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "peon", 2024-04-19T01:48:51.494 INFO:teuthology.orchestra.run.smithi062.stdout: "election_epoch": 40, 2024-04-19T01:48:51.494 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 138138, 2024-04-19T01:48:51.495 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum": [ 2024-04-19T01:48:51.495 INFO:teuthology.orchestra.run.smithi062.stdout: 0, 2024-04-19T01:48:51.495 INFO:teuthology.orchestra.run.smithi062.stdout: 1, 2024-04-19T01:48:51.495 INFO:teuthology.orchestra.run.smithi062.stdout: 2 2024-04-19T01:48:51.495 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:48:51.495 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_age": 5, 2024-04-19T01:48:51.495 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:48:51.495 INFO:teuthology.orchestra.run.smithi062.stdout: "required_con": "2450521708007866372", 2024-04-19T01:48:51.495 INFO:teuthology.orchestra.run.smithi062.stdout: "required_mon": [ 2024-04-19T01:48:51.495 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:48:51.495 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:48:51.495 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:48:51.495 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:48:51.495 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:48:51.495 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:48:51.496 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:48:51.496 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:48:51.496 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:48:51.496 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:48:51.496 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:48:51.496 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:48:51.496 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:48:51.496 INFO:teuthology.orchestra.run.smithi062.stdout: "quorum_mon": [ 2024-04-19T01:48:51.496 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:48:51.496 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:48:51.496 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:48:51.496 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:48:51.496 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:48:51.496 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:48:51.497 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:48:51.497 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:48:51.497 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:48:51.497 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:48:51.497 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:48:51.497 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:48:51.497 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:51.497 INFO:teuthology.orchestra.run.smithi062.stdout: "outside_quorum": [], 2024-04-19T01:48:51.497 INFO:teuthology.orchestra.run.smithi062.stdout: "extra_probe_peers": [], 2024-04-19T01:48:51.497 INFO:teuthology.orchestra.run.smithi062.stdout: "sync_provider": [], 2024-04-19T01:48:51.497 INFO:teuthology.orchestra.run.smithi062.stdout: "monmap": { 2024-04-19T01:48:51.497 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 1, 2024-04-19T01:48:51.497 INFO:teuthology.orchestra.run.smithi062.stdout: "fsid": "6466e4b6-2520-485f-95b3-9401915fbbad", 2024-04-19T01:48:51.497 INFO:teuthology.orchestra.run.smithi062.stdout: "modified": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:48:51.497 INFO:teuthology.orchestra.run.smithi062.stdout: "created": "2024-04-19T01:41:38.497795Z", 2024-04-19T01:48:51.498 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release": 19, 2024-04-19T01:48:51.498 INFO:teuthology.orchestra.run.smithi062.stdout: "min_mon_release_name": "squid", 2024-04-19T01:48:51.498 INFO:teuthology.orchestra.run.smithi062.stdout: "election_strategy": 1, 2024-04-19T01:48:51.498 INFO:teuthology.orchestra.run.smithi062.stdout: "disallowed_leaders: ": "", 2024-04-19T01:48:51.498 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false, 2024-04-19T01:48:51.498 INFO:teuthology.orchestra.run.smithi062.stdout: "tiebreaker_mon": "", 2024-04-19T01:48:51.498 INFO:teuthology.orchestra.run.smithi062.stdout: "removed_ranks: ": "", 2024-04-19T01:48:51.498 INFO:teuthology.orchestra.run.smithi062.stdout: "features": { 2024-04-19T01:48:51.498 INFO:teuthology.orchestra.run.smithi062.stdout: "persistent": [ 2024-04-19T01:48:51.498 INFO:teuthology.orchestra.run.smithi062.stdout: "kraken", 2024-04-19T01:48:51.498 INFO:teuthology.orchestra.run.smithi062.stdout: "luminous", 2024-04-19T01:48:51.498 INFO:teuthology.orchestra.run.smithi062.stdout: "mimic", 2024-04-19T01:48:51.498 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap-prune", 2024-04-19T01:48:51.498 INFO:teuthology.orchestra.run.smithi062.stdout: "nautilus", 2024-04-19T01:48:51.499 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus", 2024-04-19T01:48:51.499 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific", 2024-04-19T01:48:51.499 INFO:teuthology.orchestra.run.smithi062.stdout: "elector-pinging", 2024-04-19T01:48:51.499 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy", 2024-04-19T01:48:51.499 INFO:teuthology.orchestra.run.smithi062.stdout: "reef", 2024-04-19T01:48:51.499 INFO:teuthology.orchestra.run.smithi062.stdout: "squid" 2024-04-19T01:48:51.499 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:48:51.499 INFO:teuthology.orchestra.run.smithi062.stdout: "optional": [] 2024-04-19T01:48:51.499 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:51.499 INFO:teuthology.orchestra.run.smithi062.stdout: "mons": [ 2024-04-19T01:48:51.499 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:51.499 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 0, 2024-04-19T01:48:51.499 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "a", 2024-04-19T01:48:51.499 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:48:51.499 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:48:51.499 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:51.500 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:48:51.500 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3300", 2024-04-19T01:48:51.500 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:48:51.500 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:51.500 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:51.500 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:48:51.500 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789", 2024-04-19T01:48:51.500 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:48:51.500 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:48:51.500 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:48:51.500 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:51.500 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6789/0", 2024-04-19T01:48:51.500 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6789/0", 2024-04-19T01:48:51.500 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:48:51.500 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:48:51.501 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:48:51.501 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:51.501 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:51.501 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 1, 2024-04-19T01:48:51.501 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "b", 2024-04-19T01:48:51.501 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:48:51.501 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:48:51.501 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:51.501 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:48:51.501 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:3300", 2024-04-19T01:48:51.501 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:48:51.501 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:51.501 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:51.501 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:48:51.502 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789", 2024-04-19T01:48:51.502 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:48:51.502 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:48:51.502 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:48:51.502 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:51.502 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.71:6789/0", 2024-04-19T01:48:51.502 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.71:6789/0", 2024-04-19T01:48:51.502 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:48:51.502 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:48:51.502 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:48:51.502 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:51.502 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:51.502 INFO:teuthology.orchestra.run.smithi062.stdout: "rank": 2, 2024-04-19T01:48:51.502 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "c", 2024-04-19T01:48:51.503 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addrs": { 2024-04-19T01:48:51.503 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-04-19T01:48:51.503 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:51.503 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-04-19T01:48:51.503 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:3301", 2024-04-19T01:48:51.503 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:48:51.503 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:51.503 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:51.503 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-19T01:48:51.503 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790", 2024-04-19T01:48:51.503 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 0 2024-04-19T01:48:51.503 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:48:51.503 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:48:51.503 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:51.503 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6790/0", 2024-04-19T01:48:51.504 INFO:teuthology.orchestra.run.smithi062.stdout: "public_addr": "172.21.15.62:6790/0", 2024-04-19T01:48:51.504 INFO:teuthology.orchestra.run.smithi062.stdout: "priority": 0, 2024-04-19T01:48:51.504 INFO:teuthology.orchestra.run.smithi062.stdout: "weight": 0, 2024-04-19T01:48:51.504 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_location": "{}" 2024-04-19T01:48:51.504 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:48:51.504 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:48:51.504 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:51.504 INFO:teuthology.orchestra.run.smithi062.stdout: "feature_map": { 2024-04-19T01:48:51.504 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": [ 2024-04-19T01:48:51.504 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:51.504 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:48:51.504 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:48:51.504 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:48:51.504 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:48:51.505 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:48:51.505 INFO:teuthology.orchestra.run.smithi062.stdout: "client": [ 2024-04-19T01:48:51.505 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:51.505 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:48:51.505 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:48:51.505 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 2 2024-04-19T01:48:51.505 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:48:51.505 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-04-19T01:48:51.505 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": [ 2024-04-19T01:48:51.505 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-19T01:48:51.505 INFO:teuthology.orchestra.run.smithi062.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:48:51.505 INFO:teuthology.orchestra.run.smithi062.stdout: "release": "luminous", 2024-04-19T01:48:51.505 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 1 2024-04-19T01:48:51.505 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-19T01:48:51.505 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-19T01:48:51.505 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-19T01:48:51.505 INFO:teuthology.orchestra.run.smithi062.stdout: "stretch_mode": false 2024-04-19T01:48:51.506 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-19T01:48:51.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.497+0000 7f5de7c36640 1 -- 172.21.15.62:0/2897362909 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5dc4036480 msgr2=0x7f5dc4038940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:51.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.497+0000 7f5de7c36640 1 --2- 172.21.15.62:0/2897362909 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5dc4036480 0x7f5dc4038940 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f5de0126810 tx=0x7f5dd0008040 comp rx=0 tx=0).stop 2024-04-19T01:48:51.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.497+0000 7f5de7c36640 10 monclient: shutdown 2024-04-19T01:48:51.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.497+0000 7f5de7c36640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:51.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.497+0000 7f5de7c36640 1 -- 172.21.15.62:0/2897362909 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5de00a3410 msgr2=0x7f5de0125500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:51.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.497+0000 7f5de7c36640 1 --2- 172.21.15.62:0/2897362909 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5de00a3410 0x7f5de0125500 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5dd80ae000 tx=0x7f5dd8019690 comp rx=0 tx=0).stop 2024-04-19T01:48:51.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.497+0000 7f5de61d3640 1 -- 172.21.15.62:0/2897362909 reap_dead start 2024-04-19T01:48:51.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.497+0000 7f5de61d3640 1 -- 172.21.15.62:0/2897362909 reap_dead start 2024-04-19T01:48:51.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.497+0000 7f5de7c36640 1 -- 172.21.15.62:0/2897362909 shutdown_connections 2024-04-19T01:48:51.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.497+0000 7f5de7c36640 1 -- 172.21.15.62:0/2897362909 >> 172.21.15.62:0/2897362909 conn(0x7f5de00aa2d0 msgr2=0x7f5de009e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:51.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.497+0000 7f5de7c36640 1 -- 172.21.15.62:0/2897362909 shutdown_connections 2024-04-19T01:48:51.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.497+0000 7f5de7c36640 1 -- 172.21.15.62:0/2897362909 wait complete. 2024-04-19T01:48:51.508 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-19T01:48:51.508 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-19T01:48:51.509 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-19T01:48:51.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.625+0000 7ffbbe143640 10 monclient: get_monmap_and_config 2024-04-19T01:48:51.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.625+0000 7ffbbe143640 10 monclient: build_initial_monmap 2024-04-19T01:48:51.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.625+0000 7ffbbe143640 10 monclient: monmap: 2024-04-19T01:48:51.623 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:51.623 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:51.623 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:48:51.629309+0000 2024-04-19T01:48:51.623 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:48:51.629309+0000 2024-04-19T01:48:51.623 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:51.623 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:51.623 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:48:51.623 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:48:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:48:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:48:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:48:51.624 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:51.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbbe143640 1 Processor -- start 2024-04-19T01:48:51.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbbe143640 1 -- start start 2024-04-19T01:48:51.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbbe143640 10 monclient: init 2024-04-19T01:48:51.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbbe143640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:51.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbbe143640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-04-19T01:48:51.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbbe143640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffbb80a2ef0 addr v1:172.21.15.71:6789/0 2024-04-19T01:48:51.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbbe143640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffbb809d040 addr v1:172.21.15.62:6789/0 2024-04-19T01:48:51.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbbe143640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ffbb813a0e0 0x7ffbb813c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:51.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbbe143640 10 monclient(hunting): picked mon.noname-a con 0x7ffbb813a0e0 addr v2:172.21.15.62:3300/0 2024-04-19T01:48:51.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbbe143640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbb809c890 con 0x7ffbb809d040 2024-04-19T01:48:51.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbbe143640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbb809d4d0 con 0x7ffbb80a2ef0 2024-04-19T01:48:51.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbbe143640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:51.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbbe143640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7ffbb809e700 con 0x7ffbb813a0e0 2024-04-19T01:48:51.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbbe143640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:51.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbbe143640 10 monclient(hunting): authenticate will time out at 1317.448687s 2024-04-19T01:48:51.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbaffff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7ffbb809d040 0x7ffbb809c180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:56706/0 (socket says 172.21.15.62:56706) 2024-04-19T01:48:51.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbb7fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ffbb813a0e0 0x7ffbb813c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:51.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbaffff640 1 -- 172.21.15.62:0/398356360 learned_addr learned my addr 172.21.15.62:0/398356360 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:48:51.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbb7fff640 10 monclient(hunting): get_auth_request con 0x7ffbb813a0e0 auth_method 0 2024-04-19T01:48:51.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:51.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbb7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:51.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:51.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:51.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:51.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:51.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbb7fff640 10 monclient(hunting): handle_auth_done global_id 15273 payload 274 2024-04-19T01:48:51.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbb7fff640 1 -- 172.21.15.62:0/398356360 >> v1:172.21.15.71:6789/0 conn(0x7ffbb80a2ef0 legacy=0x7ffbb809ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:51.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbb7fff640 1 -- 172.21.15.62:0/398356360 >> v1:172.21.15.62:6789/0 conn(0x7ffbb809d040 legacy=0x7ffbb809c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:51.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbb7fff640 10 monclient: _finish_hunting 0 2024-04-19T01:48:51.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbb7fff640 1 monclient: found mon.noname-a 2024-04-19T01:48:51.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:51.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbb7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:48:51.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.629+0000 7ffbb7fff640 1 -- 172.21.15.62:0/398356360 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbb809d960 con 0x7ffbb813a0e0 2024-04-19T01:48:51.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbb7fff640 1 -- 172.21.15.62:0/398356360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbb813a0e0 msgr2=0x7ffbb813c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:51.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbb7fff640 1 --2- 172.21.15.62:0/398356360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbb813a0e0 0x7ffbb813c4d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0009d90 tx=0x7ffbb001c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6be18d499aab390 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:51.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbb6ffd640 1 -- 172.21.15.62:0/398356360 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb00a67d0 con 0x7ffbb813a0e0 2024-04-19T01:48:51.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbb6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:51.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbb6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:48:51.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbb6ffd640 10 monclient: dump: 2024-04-19T01:48:51.630 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:51.630 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:51.630 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:51.630 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:51.630 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:51.630 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:51.630 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:51.630 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:51.630 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:51.630 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:51.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbb6ffd640 10 monclient: _finish_auth 0 2024-04-19T01:48:51.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbb6ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:48:51.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:51.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbb6ffd640 1 -- 172.21.15.62:0/398356360 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbb00a6970 con 0x7ffbb813a0e0 2024-04-19T01:48:51.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbbe143640 5 monclient: authenticate success, global_id 15273 2024-04-19T01:48:51.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbbe143640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:48:51.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbb6ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:51.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbbe143640 10 monclient: get_monmap_and_config success 2024-04-19T01:48:51.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbb6ffd640 1 -- 172.21.15.62:0/398356360 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb00a6d40 con 0x7ffbb813a0e0 2024-04-19T01:48:51.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbb6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:51.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbbe143640 4 set_mon_vals no callback set 2024-04-19T01:48:51.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbb6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:48:51.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbb6ffd640 10 monclient: dump: 2024-04-19T01:48:51.632 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:51.632 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:51.632 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:51.632 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:51.632 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:51.632 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:51.632 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:51.632 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:51.632 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:51.632 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:51.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbbe143640 10 monclient: shutdown 2024-04-19T01:48:51.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbbe143640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:51.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbbe143640 1 -- 172.21.15.62:0/398356360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbb813a0e0 msgr2=0x7ffbb813c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:51.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbbe143640 1 --2- 172.21.15.62:0/398356360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbb813a0e0 0x7ffbb813c4d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0009d90 tx=0x7ffbb001c830 comp rx=0 tx=0).stop 2024-04-19T01:48:51.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbbe143640 1 -- 172.21.15.62:0/398356360 shutdown_connections 2024-04-19T01:48:51.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbbe143640 1 --2- 172.21.15.62:0/398356360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbb813a0e0 0x7ffbb813c4d0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:51.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbbe143640 1 -- 172.21.15.62:0/398356360 >> 172.21.15.62:0/398356360 conn(0x7ffbb80a23d0 msgr2=0x7ffbb80a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:51.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbbe143640 1 -- 172.21.15.62:0/398356360 shutdown_connections 2024-04-19T01:48:51.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbbe143640 1 -- 172.21.15.62:0/398356360 wait complete. 2024-04-19T01:48:51.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbbe143640 10 monclient: build_initial_monmap 2024-04-19T01:48:51.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbbe143640 10 monclient: monmap: 2024-04-19T01:48:51.638 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:51.638 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:48:51.638 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:48:51.638 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:48:51.638 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:51.638 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:51.638 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:48:51.638 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:48:51.638 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:48:51.638 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:51.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbbe143640 1 Processor -- start 2024-04-19T01:48:51.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbbe143640 1 -- start start 2024-04-19T01:48:51.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.633+0000 7ffbbe143640 10 monclient: init 2024-04-19T01:48:51.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbbe143640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:51.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbbe143640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-19T01:48:51.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbbe143640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffbb80a2ef0 0x7ffbb813fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:51.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbbe143640 10 monclient(hunting): picked mon.noname-b con 0x7ffbb80a2ef0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:48:51.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbbe143640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbb813a0e0 0x7ffbb8140080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:51.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbbe143640 10 monclient(hunting): picked mon.noname-a con 0x7ffbb813a0e0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:48:51.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbbe143640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffbb81405c0 0x7ffbb81d5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:51.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbbe143640 10 monclient(hunting): picked mon.noname-c con 0x7ffbb81405c0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:51.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbbe143640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:51.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbbe143640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb813eba0 con 0x7ffbb813a0e0 2024-04-19T01:48:51.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbbe143640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:51.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbbe143640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb813ea20 con 0x7ffbb80a2ef0 2024-04-19T01:48:51.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbbe143640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:51.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbbe143640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7ffbb813ed20 con 0x7ffbb81405c0 2024-04-19T01:48:51.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbbe143640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:51.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbbe143640 10 monclient(hunting): authenticate will time out at 1317.456327s 2024-04-19T01:48:51.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbaffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbb813a0e0 0x7ffbb8140080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:51.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbaffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbb813a0e0 0x7ffbb8140080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47760/0 (socket says 172.21.15.62:47760) 2024-04-19T01:48:51.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbaffff640 1 -- 172.21.15.62:0/360275038 learned_addr learned my addr 172.21.15.62:0/360275038 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:51.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbaffff640 10 monclient(hunting): get_auth_request con 0x7ffbb813a0e0 auth_method 0 2024-04-19T01:48:51.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:51.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbaffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:51.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbaffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:51.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbb7fff640 1 --2- 172.21.15.62:0/360275038 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffbb81405c0 0x7ffbb81d5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:51.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbaffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:51.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbaffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:51.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbaffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:51.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbb7fff640 10 monclient(hunting): get_auth_request con 0x7ffbb81405c0 auth_method 0 2024-04-19T01:48:51.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:51.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbb7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:51.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:51.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbb77fe640 1 --2- 172.21.15.62:0/360275038 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffbb80a2ef0 0x7ffbb813fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:51.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbb77fe640 1 -- 172.21.15.62:0/360275038 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffbb80a2ef0 msgr2=0x7ffbb813fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-19T01:48:51.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbb77fe640 1 -- 172.21.15.62:0/360275038 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffbb80a2ef0 msgr2=0x7ffbb813fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:48:51.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbb77fe640 1 --2- 172.21.15.62:0/360275038 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffbb80a2ef0 0x7ffbb813fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:48:51.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:51.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:51.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbb77fe640 1 --2- 172.21.15.62:0/360275038 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffbb80a2ef0 0x7ffbb813fb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:48:51.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:51.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbaffff640 10 monclient(hunting): handle_auth_done global_id 15276 payload 824 2024-04-19T01:48:51.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbaffff640 1 -- 172.21.15.62:0/360275038 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffbb81405c0 msgr2=0x7ffbb81d5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:51.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbaffff640 1 --2- 172.21.15.62:0/360275038 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffbb81405c0 0x7ffbb81d5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:51.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbaffff640 1 -- 172.21.15.62:0/360275038 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffbb80a2ef0 msgr2=0x7ffbb813fb40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:48:51.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbaffff640 1 --2- 172.21.15.62:0/360275038 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffbb80a2ef0 0x7ffbb813fb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:51.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbaffff640 10 monclient: _finish_hunting 0 2024-04-19T01:48:51.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbaffff640 1 monclient: found mon.noname-a 2024-04-19T01:48:51.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbaffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:51.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbaffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:48:51.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbaffff640 1 -- 172.21.15.62:0/360275038 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbb811bfc0 con 0x7ffbb813a0e0 2024-04-19T01:48:51.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbaffff640 1 -- 172.21.15.62:0/360275038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbb813a0e0 msgr2=0x7ffbb8140080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:51.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbaffff640 1 --2- 172.21.15.62:0/360275038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbb813a0e0 0x7ffbb8140080 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7ffba000b430 tx=0x7ffba000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:51.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbb4ff9640 1 -- 172.21.15.62:0/360275038 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba0015020 con 0x7ffbb813a0e0 2024-04-19T01:48:51.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbb4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:51.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbb4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:48:51.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbb4ff9640 10 monclient: dump: 2024-04-19T01:48:51.642 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:51.642 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:51.642 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:51.642 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:51.642 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:51.642 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:51.643 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:51.643 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:51.643 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:51.643 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:51.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbb4ff9640 10 monclient: _finish_auth 0 2024-04-19T01:48:51.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbb4ff9640 10 monclient: _check_auth_tickets 2024-04-19T01:48:51.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbb4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:51.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbb4ff9640 1 -- 172.21.15.62:0/360275038 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffba0011a60 con 0x7ffbb813a0e0 2024-04-19T01:48:51.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbbe143640 5 monclient: authenticate success, global_id 15276 2024-04-19T01:48:51.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbbe143640 10 monclient: _renew_subs 2024-04-19T01:48:51.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbbe143640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:51.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbbe143640 1 -- 172.21.15.62:0/360275038 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbb80729b0 con 0x7ffbb813a0e0 2024-04-19T01:48:51.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbb4ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:51.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbb4ff9640 1 -- 172.21.15.62:0/360275038 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba0011e50 con 0x7ffbb813a0e0 2024-04-19T01:48:51.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbb4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:51.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbb4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:48:51.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbb4ff9640 10 monclient: dump: 2024-04-19T01:48:51.644 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:51.644 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:51.644 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:51.644 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:51.644 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:51.644 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:51.644 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:51.644 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:51.644 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:51.644 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:51.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbbe143640 10 monclient: _renew_subs 2024-04-19T01:48:51.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbb67fc640 4 set_mon_vals no callback set 2024-04-19T01:48:51.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbbe143640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:51.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.637+0000 7ffbbe143640 1 -- 172.21.15.62:0/360275038 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbb8141820 con 0x7ffbb813a0e0 2024-04-19T01:48:51.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.641+0000 7ffbbe143640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:51.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.641+0000 7ffbbe143640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:51.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.641+0000 7ffbbe143640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:51.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.641+0000 7ffbb4ff9640 1 -- 172.21.15.62:0/360275038 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7ffba0018020 con 0x7ffbb813a0e0 2024-04-19T01:48:51.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.641+0000 7ffbbe143640 1 -- 172.21.15.62:0/360275038 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbb80692b0 con 0x7ffbb813a0e0 2024-04-19T01:48:51.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.641+0000 7ffbb4ff9640 1 --2- 172.21.15.62:0/360275038 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7ffb98036610 0x7ffb98038ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:51.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.641+0000 7ffbb4ff9640 1 -- 172.21.15.62:0/360275038 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(325..325 src has 115..325) ==== 4346+0+0 (secure 0 0 0) 0x7ffba0049260 con 0x7ffbb813a0e0 2024-04-19T01:48:51.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.641+0000 7ffbb77fe640 1 --2- 172.21.15.62:0/360275038 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7ffb98036610 0x7ffb98038ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:51.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.641+0000 7ffbb77fe640 10 monclient: get_auth_request con 0x7ffb98036610 auth_method 0 2024-04-19T01:48:51.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.641+0000 7ffbb77fe640 1 --2- 172.21.15.62:0/360275038 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7ffb98036610 0x7ffb98038ad0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ffba8007870 tx=0x7ffba8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:51.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.645+0000 7ffbb4ff9640 1 -- 172.21.15.62:0/360275038 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ffba001f020 con 0x7ffbb813a0e0 2024-04-19T01:48:51.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.645+0000 7ffbb4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:51.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.645+0000 7ffbb4ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:48:51.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.869+0000 7ffbbe143640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-19T01:48:51.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.869+0000 7ffbbe143640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-19T01:48:51.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.869+0000 7ffbbe143640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:51.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.869+0000 7ffbbe143640 1 -- 172.21.15.62:0/360275038 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7ffbb8139a40 con 0x7ffbb813a0e0 2024-04-19T01:48:51.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.873+0000 7ffbb4ff9640 1 -- 172.21.15.62:0/360275038 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7ffba0047e10 con 0x7ffbb813a0e0 2024-04-19T01:48:51.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.873+0000 7ffbb4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-19T01:48:51.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.873+0000 7ffbb4ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:48:51.872 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:51.877+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:51.872 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:51.877+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=11,logm=65} crc {auth=413674553,config=3541143443,health=1535303502,kv=3416605158,logm=973187141}) 2024-04-19T01:48:51.872 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:51.877+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=12,kv=11,logm=65} crc {auth=413674553,config=3541143443,health=1535303502,kv=3416605158,logm=3523822805}) 2024-04-19T01:48:51.872 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:51.877+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:51.872 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:51.877+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=11,logm=65} crc {auth=413674553,config=3541143443,health=1535303502,kv=3416605158,logm=973187141}) 2024-04-19T01:48:51.872 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:51.877+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=2,health=12,kv=11,logm=65} crc {auth=413674553,config=3541143443,health=1535303502,kv=3416605158,logm=1989569898}) 2024-04-19T01:48:51.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.877+0000 7ffbbe143640 1 -- 172.21.15.62:0/360275038 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7ffb98036610 msgr2=0x7ffb98038ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:51.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.877+0000 7ffbbe143640 1 --2- 172.21.15.62:0/360275038 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7ffb98036610 0x7ffb98038ad0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ffba8007870 tx=0x7ffba8009040 comp rx=0 tx=0).stop 2024-04-19T01:48:51.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.877+0000 7ffbbe143640 10 monclient: shutdown 2024-04-19T01:48:51.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.877+0000 7ffbbe143640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:51.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.877+0000 7ffbbe143640 1 -- 172.21.15.62:0/360275038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbb813a0e0 msgr2=0x7ffbb8140080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:51.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.877+0000 7ffbbe143640 1 --2- 172.21.15.62:0/360275038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbb813a0e0 0x7ffbb8140080 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7ffba000b430 tx=0x7ffba000b8f0 comp rx=0 tx=0).stop 2024-04-19T01:48:51.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.877+0000 7ffbbe143640 1 -- 172.21.15.62:0/360275038 shutdown_connections 2024-04-19T01:48:51.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.877+0000 7ffbbe143640 1 --2- 172.21.15.62:0/360275038 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7ffb98036610 0x7ffb98038ad0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:51.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.877+0000 7ffbbe143640 1 --2- 172.21.15.62:0/360275038 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffbb81405c0 0x7ffbb81d5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:51.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.877+0000 7ffbbe143640 1 --2- 172.21.15.62:0/360275038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbb813a0e0 0x7ffbb8140080 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:51.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.877+0000 7ffbbe143640 1 --2- 172.21.15.62:0/360275038 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffbb80a2ef0 0x7ffbb813fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:51.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.877+0000 7ffbbe143640 1 -- 172.21.15.62:0/360275038 >> 172.21.15.62:0/360275038 conn(0x7ffbb80a23d0 msgr2=0x7ffbb813a670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:51.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.877+0000 7ffbbe143640 1 -- 172.21.15.62:0/360275038 shutdown_connections 2024-04-19T01:48:51.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:51.877+0000 7ffbbe143640 1 -- 172.21.15.62:0/360275038 wait complete. 2024-04-19T01:48:51.890 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-19T01:48:52.058 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.061+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.058 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.061+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2112946676}) 2024-04-19T01:48:52.058 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.061+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4049383318}) 2024-04-19T01:48:52.058 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.061+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.058 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.061+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2112946676}) 2024-04-19T01:48:52.058 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.061+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1829868511}) 2024-04-19T01:48:52.060 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.065+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.060 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.065+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1801574747}) 2024-04-19T01:48:52.060 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.065+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=609202812}) 2024-04-19T01:48:52.060 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.065+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.060 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.065+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1801574747}) 2024-04-19T01:48:52.060 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.065+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1100636853}) 2024-04-19T01:48:52.062 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.065+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.062 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.065+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3449663772}) 2024-04-19T01:48:52.062 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.065+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=744482626}) 2024-04-19T01:48:52.063 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.065+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.063 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.065+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3449663772}) 2024-04-19T01:48:52.063 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.065+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2006326522}) 2024-04-19T01:48:52.065 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.069+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.065 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.069+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=74516090}) 2024-04-19T01:48:52.065 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.069+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1216113639}) 2024-04-19T01:48:52.065 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.069+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.065 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.069+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=74516090}) 2024-04-19T01:48:52.065 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.069+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3378267560}) 2024-04-19T01:48:52.071 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.069+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.071 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.069+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3637385751}) 2024-04-19T01:48:52.071 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.069+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2000580522}) 2024-04-19T01:48:52.071 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.069+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.071 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.069+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3637385751}) 2024-04-19T01:48:52.071 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.069+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=373008991}) 2024-04-19T01:48:52.071 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.073+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.071 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.073+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4138119681}) 2024-04-19T01:48:52.071 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.073+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2742707138}) 2024-04-19T01:48:52.071 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.073+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.071 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.073+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4138119681}) 2024-04-19T01:48:52.071 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.073+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3700441376}) 2024-04-19T01:48:52.072 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.077+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.073 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.077+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3729456153}) 2024-04-19T01:48:52.073 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.077+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2547754705}) 2024-04-19T01:48:52.073 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.077+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.073 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.077+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3729456153}) 2024-04-19T01:48:52.073 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.077+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2275172977}) 2024-04-19T01:48:52.075 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.077+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.075 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.077+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=27293816}) 2024-04-19T01:48:52.075 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.077+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1535806177}) 2024-04-19T01:48:52.075 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.077+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.075 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.077+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=27293816}) 2024-04-19T01:48:52.075 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.077+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3760766670}) 2024-04-19T01:48:52.078 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.081+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.078 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.081+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3865886942}) 2024-04-19T01:48:52.078 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.081+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3576318317}) 2024-04-19T01:48:52.078 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.081+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.078 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.081+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3865886942}) 2024-04-19T01:48:52.078 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.081+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2291194687}) 2024-04-19T01:48:52.080 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.085+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.081 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.085+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3197181955}) 2024-04-19T01:48:52.081 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.085+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1642375704}) 2024-04-19T01:48:52.081 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.085+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.081 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.085+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3197181955}) 2024-04-19T01:48:52.081 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.085+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1385959368}) 2024-04-19T01:48:52.083 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.085+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.084 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.085+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2275060172}) 2024-04-19T01:48:52.084 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.085+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2248573701}) 2024-04-19T01:48:52.084 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.085+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.084 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.085+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2275060172}) 2024-04-19T01:48:52.084 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.085+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3708091210}) 2024-04-19T01:48:52.086 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.089+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.086 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.089+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2573149650}) 2024-04-19T01:48:52.086 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.089+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3998335643}) 2024-04-19T01:48:52.086 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.089+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.086 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.089+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2573149650}) 2024-04-19T01:48:52.086 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.089+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1742664942}) 2024-04-19T01:48:52.088 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.093+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.088 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.093+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3517611382}) 2024-04-19T01:48:52.089 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.093+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3972553618}) 2024-04-19T01:48:52.089 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.093+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.089 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.093+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3517611382}) 2024-04-19T01:48:52.089 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.093+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1454537337}) 2024-04-19T01:48:52.099 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.101+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.099 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.101+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=67,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=5,mon_config_key=12,monmap=3} crc {logm=925375995,mdsmap=2840952586,mgr=3717952716,mgr_command_descs=3636332126,mgr_metadata=230233602,mgrstat=2748063499,mon_config_key=2742521082,monmap=35495450}) 2024-04-19T01:48:52.100 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.101+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=66,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=14,monmap=2} crc {logm=2895787396,mdsmap=3140514009,mgr=3531289824,mgr_command_descs=3636332126,mgr_metadata=230233602,mgrstat=4020818467,mon_config_key=1430845806,monmap=405663308}) 2024-04-19T01:48:52.100 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.101+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.100 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.101+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=67,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=5,mon_config_key=12,monmap=3} crc {logm=925375995,mdsmap=2840952586,mgr=3717952716,mgr_command_descs=3636332126,mgr_metadata=230233602,mgrstat=2748063499,mon_config_key=2742521082,monmap=35495450}) 2024-04-19T01:48:52.100 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.101+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=10} crc {logm=1402419152,mdsmap=2840952586,mgr=3717952716,mgr_command_descs=3636332126,mgr_metadata=230233602,mgrstat=2413793058,mon_config_key=1189232275}) 2024-04-19T01:48:52.101 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.105+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.102 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.105+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=4,osd_pg_creating=1,osdmap=95} crc {osd_metadata=2955798063,osd_pg_creating=1729266415,osdmap=248198391}) 2024-04-19T01:48:52.102 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.105+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_metadata=6,osd_pg_creating=1,osdmap=93} crc {osd_metadata=2171542278,osd_pg_creating=1729266415,osdmap=3261256634}) 2024-04-19T01:48:52.102 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.105+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.102 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.105+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=4,osd_pg_creating=1,osdmap=95} crc {osd_metadata=2955798063,osd_pg_creating=1729266415,osdmap=248198391}) 2024-04-19T01:48:52.102 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.105+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_metadata=6,osd_pg_creating=1,osdmap=93} crc {osd_metadata=2171542278,osd_pg_creating=1729266415,osdmap=3791918597}) 2024-04-19T01:48:52.224 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.229+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.224 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.229+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1847183917}) 2024-04-19T01:48:52.224 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.229+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1666078340}) 2024-04-19T01:48:52.224 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.229+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.224 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.229+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1847183917}) 2024-04-19T01:48:52.224 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.229+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4162823573}) 2024-04-19T01:48:52.227 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.229+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.227 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.229+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2983343494}) 2024-04-19T01:48:52.227 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.229+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1828679676}) 2024-04-19T01:48:52.227 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.229+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.227 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.229+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2983343494}) 2024-04-19T01:48:52.227 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.229+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1586537967}) 2024-04-19T01:48:52.229 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.233+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:48:52.229 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.233+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=23} crc {osdmap=3840804011}) 2024-04-19T01:48:52.229 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-19T01:48:52.233+0000 7fd9ebe71640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=24} crc {osdmap=3843305711}) 2024-04-19T01:48:52.890 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-19T01:48:52.890 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:48:53.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf40e5b640 10 monclient: get_monmap_and_config 2024-04-19T01:48:53.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf40e5b640 10 monclient: build_initial_monmap 2024-04-19T01:48:53.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf40e5b640 10 monclient: monmap: 2024-04-19T01:48:53.021 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:53.021 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:53.021 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:48:53.027094+0000 2024-04-19T01:48:53.021 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:48:53.027094+0000 2024-04-19T01:48:53.021 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:53.021 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:53.021 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:48:53.021 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:48:53.021 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:48:53.021 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:48:53.021 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:48:53.022 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:53.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf40e5b640 1 Processor -- start 2024-04-19T01:48:53.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf40e5b640 1 -- start start 2024-04-19T01:48:53.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf40e5b640 10 monclient: init 2024-04-19T01:48:53.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf40e5b640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:53.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf40e5b640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-04-19T01:48:53.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf40e5b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcf3c0a47e0 0x7fcf3c0a4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:53.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf40e5b640 10 monclient(hunting): picked mon.noname-a con 0x7fcf3c0a47e0 addr v2:172.21.15.62:3300/0 2024-04-19T01:48:53.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf40e5b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcf3c0a5100 addr v1:172.21.15.62:6789/0 2024-04-19T01:48:53.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf40e5b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fcf3c1357c0 0x7fcf3c13d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:53.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf40e5b640 10 monclient(hunting): picked mon.noname-c con 0x7fcf3c1357c0 addr v2:172.21.15.71:3300/0 2024-04-19T01:48:53.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf40e5b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf3c09f7c0 con 0x7fcf3c0a5100 2024-04-19T01:48:53.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf40e5b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:53.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf40e5b640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fcf3c09ee90 con 0x7fcf3c0a47e0 2024-04-19T01:48:53.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf40e5b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:53.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf40e5b640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fcf3c09c2d0 con 0x7fcf3c1357c0 2024-04-19T01:48:53.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf40e5b640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:53.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf40e5b640 10 monclient(hunting): authenticate will time out at 1318.847072s 2024-04-19T01:48:53.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf3a59c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcf3c0a47e0 0x7fcf3c0a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:53.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf33fff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fcf3c0a5100 0x7fcf3c1350b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:56716/0 (socket says 172.21.15.62:56716) 2024-04-19T01:48:53.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf33fff640 1 -- 172.21.15.62:0/2377758485 learned_addr learned my addr 172.21.15.62:0/2377758485 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:48:53.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf3a59c640 1 --2- 172.21.15.62:0/2377758485 >> v2:172.21.15.62:3300/0 conn(0x7fcf3c0a47e0 0x7fcf3c0a4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47776/0 (socket says 172.21.15.62:47776) 2024-04-19T01:48:53.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf3a59c640 10 monclient(hunting): get_auth_request con 0x7fcf3c0a47e0 auth_method 0 2024-04-19T01:48:53.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf3a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:53.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf3a59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:53.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf3a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:53.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf3a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:53.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf3a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:53.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf3a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:53.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf3ad9d640 1 --2- 172.21.15.62:0/2377758485 >> v2:172.21.15.71:3300/0 conn(0x7fcf3c1357c0 0x7fcf3c13d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:53.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf3ad9d640 10 monclient(hunting): get_auth_request con 0x7fcf3c1357c0 auth_method 0 2024-04-19T01:48:53.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf3ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:53.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf3ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:53.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf3ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:53.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf3a59c640 10 monclient(hunting): handle_auth_done global_id 15279 payload 274 2024-04-19T01:48:53.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf3a59c640 1 -- 172.21.15.62:0/2377758485 >> v2:172.21.15.71:3300/0 conn(0x7fcf3c1357c0 msgr2=0x7fcf3c13d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:53.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf3a59c640 1 --2- 172.21.15.62:0/2377758485 >> v2:172.21.15.71:3300/0 conn(0x7fcf3c1357c0 0x7fcf3c13d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:53.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf3a59c640 1 -- 172.21.15.62:0/2377758485 >> v1:172.21.15.62:6789/0 conn(0x7fcf3c0a5100 legacy=0x7fcf3c1350b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:53.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf3a59c640 10 monclient: _finish_hunting 0 2024-04-19T01:48:53.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf3a59c640 1 monclient: found mon.noname-a 2024-04-19T01:48:53.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf3a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:53.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf3a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:48:53.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.025+0000 7fcf3a59c640 1 -- 172.21.15.62:0/2377758485 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf3c09faa0 con 0x7fcf3c0a47e0 2024-04-19T01:48:53.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.029+0000 7fcf3a59c640 1 -- 172.21.15.62:0/2377758485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf3c0a47e0 msgr2=0x7fcf3c0a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:53.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.029+0000 7fcf3a59c640 1 --2- 172.21.15.62:0/2377758485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf3c0a47e0 0x7fcf3c0a4bc0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c0097c0 tx=0x7fcf2c01c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19e1d9020d551cc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:53.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf39d9b640 1 -- 172.21.15.62:0/2377758485 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c0b0070 con 0x7fcf3c0a47e0 2024-04-19T01:48:53.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf39d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:53.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf39d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:48:53.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf39d9b640 10 monclient: dump: 2024-04-19T01:48:53.029 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:53.029 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:53.029 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:53.029 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:53.029 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:53.029 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:53.029 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:53.029 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:53.029 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:53.029 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:53.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf39d9b640 10 monclient: _finish_auth 0 2024-04-19T01:48:53.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf39d9b640 10 monclient: _check_auth_tickets 2024-04-19T01:48:53.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf39d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:53.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf39d9b640 1 -- 172.21.15.62:0/2377758485 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf2c0ab600 con 0x7fcf3c0a47e0 2024-04-19T01:48:53.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf39d9b640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:53.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf39d9b640 1 -- 172.21.15.62:0/2377758485 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c0ab9d0 con 0x7fcf3c0a47e0 2024-04-19T01:48:53.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 5 monclient: authenticate success, global_id 15279 2024-04-19T01:48:53.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 10 monclient: get_monmap_and_config success 2024-04-19T01:48:53.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 4 set_mon_vals no callback set 2024-04-19T01:48:53.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf39d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:53.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf39d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:48:53.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf39d9b640 10 monclient: dump: 2024-04-19T01:48:53.030 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:53.030 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:53.030 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:53.030 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:53.030 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:53.031 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:53.031 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:53.031 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:53.031 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:53.031 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:53.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 10 monclient: shutdown 2024-04-19T01:48:53.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:53.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 1 -- 172.21.15.62:0/2377758485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf3c0a47e0 msgr2=0x7fcf3c0a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:53.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 1 --2- 172.21.15.62:0/2377758485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf3c0a47e0 0x7fcf3c0a4bc0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c0097c0 tx=0x7fcf2c01c260 comp rx=0 tx=0).stop 2024-04-19T01:48:53.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 1 -- 172.21.15.62:0/2377758485 shutdown_connections 2024-04-19T01:48:53.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 1 --2- 172.21.15.62:0/2377758485 >> v2:172.21.15.71:3300/0 conn(0x7fcf3c1357c0 0x7fcf3c13d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:53.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 1 --2- 172.21.15.62:0/2377758485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf3c0a47e0 0x7fcf3c0a4bc0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:53.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 1 -- 172.21.15.62:0/2377758485 >> 172.21.15.62:0/2377758485 conn(0x7fcf3c0a23d0 msgr2=0x7fcf3c0a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:53.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 1 -- 172.21.15.62:0/2377758485 shutdown_connections 2024-04-19T01:48:53.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 1 -- 172.21.15.62:0/2377758485 wait complete. 2024-04-19T01:48:53.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 10 monclient: build_initial_monmap 2024-04-19T01:48:53.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 10 monclient: monmap: 2024-04-19T01:48:53.035 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:53.035 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:48:53.036 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:48:53.036 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:48:53.036 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:53.036 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:53.036 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:48:53.036 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:48:53.036 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:48:53.036 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:53.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 1 Processor -- start 2024-04-19T01:48:53.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 1 -- start start 2024-04-19T01:48:53.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 10 monclient: init 2024-04-19T01:48:53.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:53.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-19T01:48:53.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf3c0a47e0 0x7fcf3c1cfd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:53.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 10 monclient(hunting): picked mon.noname-b con 0x7fcf3c0a47e0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:48:53.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf3c0a5100 0x7fcf3c1d0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:53.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 10 monclient(hunting): picked mon.noname-c con 0x7fcf3c0a5100 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:53.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf3c1357c0 0x7fcf3c1cc340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:53.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 10 monclient(hunting): picked mon.noname-a con 0x7fcf3c1357c0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:48:53.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:53.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fcf3c09c8e0 con 0x7fcf3c1357c0 2024-04-19T01:48:53.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:53.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fcf3c09c760 con 0x7fcf3c0a47e0 2024-04-19T01:48:53.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:53.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fcf3c140360 con 0x7fcf3c0a5100 2024-04-19T01:48:53.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:53.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf40e5b640 10 monclient(hunting): authenticate will time out at 1318.854462s 2024-04-19T01:48:53.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf3ad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf3c1357c0 0x7fcf3c1cc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:53.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf3ad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf3c1357c0 0x7fcf3c1cc340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47784/0 (socket says 172.21.15.62:47784) 2024-04-19T01:48:53.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf3ad9d640 1 -- 172.21.15.62:0/1297822862 learned_addr learned my addr 172.21.15.62:0/1297822862 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:53.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf3ad9d640 10 monclient(hunting): get_auth_request con 0x7fcf3c1357c0 auth_method 0 2024-04-19T01:48:53.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf33fff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf3c0a5100 0x7fcf3c1d0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:53.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf3ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:53.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf3ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:53.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf3ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:53.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf33fff640 10 monclient(hunting): get_auth_request con 0x7fcf3c0a5100 auth_method 0 2024-04-19T01:48:53.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:53.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf33fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:53.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf33fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:53.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf3ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:53.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf3ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:53.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf3ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:53.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:53.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:53.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:53.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf3a59c640 1 --2- 172.21.15.62:0/1297822862 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf3c0a47e0 0x7fcf3c1cfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:53.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf3ad9d640 10 monclient(hunting): handle_auth_done global_id 15282 payload 824 2024-04-19T01:48:53.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf3ad9d640 1 -- 172.21.15.62:0/1297822862 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf3c0a5100 msgr2=0x7fcf3c1d0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:53.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf3ad9d640 1 --2- 172.21.15.62:0/1297822862 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf3c0a5100 0x7fcf3c1d0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:53.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf3ad9d640 1 -- 172.21.15.62:0/1297822862 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf3c0a47e0 msgr2=0x7fcf3c1cfd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:53.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf3ad9d640 1 --2- 172.21.15.62:0/1297822862 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf3c0a47e0 0x7fcf3c1cfd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:53.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf3ad9d640 10 monclient: _finish_hunting 0 2024-04-19T01:48:53.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf3ad9d640 1 monclient: found mon.noname-a 2024-04-19T01:48:53.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf3ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:53.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf3ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:48:53.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf3ad9d640 1 -- 172.21.15.62:0/1297822862 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf3c1d07d0 con 0x7fcf3c1357c0 2024-04-19T01:48:53.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf3a59c640 10 monclient: get_auth_request con 0x7fcf3c0a47e0 auth_method 0 2024-04-19T01:48:53.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.033+0000 7fcf3a59c640 1 --2- 172.21.15.62:0/1297822862 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf3c0a47e0 0x7fcf3c1cfd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:48:53.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.037+0000 7fcf3ad9d640 1 -- 172.21.15.62:0/1297822862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf3c1357c0 msgr2=0x7fcf3c1cc340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:53.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.037+0000 7fcf3ad9d640 1 --2- 172.21.15.62:0/1297822862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf3c1357c0 0x7fcf3c1cc340 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fcf34007e40 tx=0x7fcf3400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:53.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.037+0000 7fcf32ffd640 1 -- 172.21.15.62:0/1297822862 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf34013790 con 0x7fcf3c1357c0 2024-04-19T01:48:53.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.037+0000 7fcf32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:53.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.037+0000 7fcf32ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:48:53.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.037+0000 7fcf32ffd640 10 monclient: dump: 2024-04-19T01:48:53.040 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:53.040 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:53.040 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:53.040 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:53.040 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:53.040 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:53.040 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:53.041 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:53.041 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:53.041 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:53.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.037+0000 7fcf32ffd640 10 monclient: _finish_auth 0 2024-04-19T01:48:53.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.037+0000 7fcf32ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:48:53.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.037+0000 7fcf32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:53.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.037+0000 7fcf32ffd640 1 -- 172.21.15.62:0/1297822862 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf34019070 con 0x7fcf3c1357c0 2024-04-19T01:48:53.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.037+0000 7fcf32ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:53.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.037+0000 7fcf32ffd640 1 -- 172.21.15.62:0/1297822862 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf34013b90 con 0x7fcf3c1357c0 2024-04-19T01:48:53.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.037+0000 7fcf40e5b640 5 monclient: authenticate success, global_id 15282 2024-04-19T01:48:53.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.037+0000 7fcf40e5b640 10 monclient: _renew_subs 2024-04-19T01:48:53.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.037+0000 7fcf40e5b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:53.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.037+0000 7fcf40e5b640 1 -- 172.21.15.62:0/1297822862 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf3c0726d0 con 0x7fcf3c1357c0 2024-04-19T01:48:53.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.037+0000 7fcf32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:53.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.037+0000 7fcf3959a640 4 set_mon_vals no callback set 2024-04-19T01:48:53.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.037+0000 7fcf32ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:48:53.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.037+0000 7fcf32ffd640 10 monclient: dump: 2024-04-19T01:48:53.042 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:53.042 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:53.042 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:53.042 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:53.042 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:53.042 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:53.042 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:53.042 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:53.042 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:53.042 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:53.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.037+0000 7fcf40e5b640 10 monclient: _renew_subs 2024-04-19T01:48:53.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.037+0000 7fcf40e5b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:53.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.037+0000 7fcf40e5b640 1 -- 172.21.15.62:0/1297822862 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf3c13bb50 con 0x7fcf3c1357c0 2024-04-19T01:48:53.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.037+0000 7fcf40e5b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:53.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.037+0000 7fcf40e5b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:53.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.037+0000 7fcf40e5b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:53.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.037+0000 7fcf40e5b640 1 -- 172.21.15.62:0/1297822862 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf3c0692b0 con 0x7fcf3c1357c0 2024-04-19T01:48:53.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.037+0000 7fcf32ffd640 1 -- 172.21.15.62:0/1297822862 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fcf3401e460 con 0x7fcf3c1357c0 2024-04-19T01:48:53.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.037+0000 7fcf32ffd640 1 --2- 172.21.15.62:0/1297822862 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fcf1c0364d0 0x7fcf1c038990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:53.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.037+0000 7fcf32ffd640 1 -- 172.21.15.62:0/1297822862 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(325..325 src has 115..325) ==== 4346+0+0 (secure 0 0 0) 0x7fcf34049b20 con 0x7fcf3c1357c0 2024-04-19T01:48:53.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.037+0000 7fcf3a59c640 1 --2- 172.21.15.62:0/1297822862 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fcf1c0364d0 0x7fcf1c038990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:53.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.037+0000 7fcf3a59c640 10 monclient: get_auth_request con 0x7fcf1c0364d0 auth_method 0 2024-04-19T01:48:53.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.041+0000 7fcf3a59c640 1 --2- 172.21.15.62:0/1297822862 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fcf1c0364d0 0x7fcf1c038990 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c0ab000 tx=0x7fcf2c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:53.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.041+0000 7fcf32ffd640 1 -- 172.21.15.62:0/1297822862 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fcf3c1d07d0 con 0x7fcf3c1357c0 2024-04-19T01:48:53.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.041+0000 7fcf32ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:53.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.041+0000 7fcf32ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:48:53.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.269+0000 7fcf40e5b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:48:53.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.269+0000 7fcf40e5b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:48:53.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.269+0000 7fcf40e5b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:53.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.269+0000 7fcf40e5b640 1 -- 172.21.15.62:0/1297822862 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcf3c06b980 con 0x7fcf3c1357c0 2024-04-19T01:48:53.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.269+0000 7fcf32ffd640 1 -- 172.21.15.62:0/1297822862 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fcf34013d30 con 0x7fcf3c1357c0 2024-04-19T01:48:53.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.269+0000 7fcf32ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:48:53.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.269+0000 7fcf32ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:48:53.267 INFO:teuthology.orchestra.run.smithi062.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","modified":"2024-04-19T01:41:38.497795Z","created":"2024-04-19T01:41:38.497795Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:48:53.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.273+0000 7fcf40e5b640 1 -- 172.21.15.62:0/1297822862 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fcf1c0364d0 msgr2=0x7fcf1c038990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:53.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.273+0000 7fcf40e5b640 1 --2- 172.21.15.62:0/1297822862 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fcf1c0364d0 0x7fcf1c038990 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c0ab000 tx=0x7fcf2c008040 comp rx=0 tx=0).stop 2024-04-19T01:48:53.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.273+0000 7fcf40e5b640 10 monclient: shutdown 2024-04-19T01:48:53.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.273+0000 7fcf40e5b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:53.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.273+0000 7fcf40e5b640 1 -- 172.21.15.62:0/1297822862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf3c1357c0 msgr2=0x7fcf3c1cc340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:53.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.273+0000 7fcf40e5b640 1 --2- 172.21.15.62:0/1297822862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf3c1357c0 0x7fcf3c1cc340 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fcf34007e40 tx=0x7fcf3400e4b0 comp rx=0 tx=0).stop 2024-04-19T01:48:53.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.273+0000 7fcf40e5b640 1 -- 172.21.15.62:0/1297822862 shutdown_connections 2024-04-19T01:48:53.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.273+0000 7fcf40e5b640 1 --2- 172.21.15.62:0/1297822862 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fcf1c0364d0 0x7fcf1c038990 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:53.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.273+0000 7fcf40e5b640 1 --2- 172.21.15.62:0/1297822862 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fcf3c0a5100 0x7fcf3c1d0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:53.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.273+0000 7fcf40e5b640 1 --2- 172.21.15.62:0/1297822862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf3c1357c0 0x7fcf3c1cc340 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:53.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.273+0000 7fcf40e5b640 1 --2- 172.21.15.62:0/1297822862 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf3c0a47e0 0x7fcf3c1cfd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:53.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.273+0000 7fcf40e5b640 1 -- 172.21.15.62:0/1297822862 >> 172.21.15.62:0/1297822862 conn(0x7fcf3c0a23d0 msgr2=0x7fcf3c0a11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:53.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.273+0000 7fcf40e5b640 1 -- 172.21.15.62:0/1297822862 shutdown_connections 2024-04-19T01:48:53.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.273+0000 7fcf40e5b640 1 -- 172.21.15.62:0/1297822862 wait complete. 2024-04-19T01:48:53.285 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-19T01:48:53.286 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-19T01:48:53.296 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-19T01:48:53.296 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:48:53.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a0c876640 10 monclient: get_monmap_and_config 2024-04-19T01:48:53.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a0c876640 10 monclient: build_initial_monmap 2024-04-19T01:48:53.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a0c876640 10 monclient: monmap: 2024-04-19T01:48:53.424 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:53.424 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:53.425 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:48:53.430566+0000 2024-04-19T01:48:53.425 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:48:53.430566+0000 2024-04-19T01:48:53.425 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:53.425 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:53.425 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:48:53.425 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:48:53.425 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:48:53.425 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:48:53.425 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:48:53.425 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:53.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a0c876640 1 Processor -- start 2024-04-19T01:48:53.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a0c876640 1 -- start start 2024-04-19T01:48:53.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a0c876640 10 monclient: init 2024-04-19T01:48:53.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a0c876640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:53.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a0c876640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-04-19T01:48:53.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a0c876640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0a080a47e0 0x7f0a080a4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:53.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a0c876640 10 monclient(hunting): picked mon.noname-a con 0x7f0a080a47e0 addr v2:172.21.15.62:3300/0 2024-04-19T01:48:53.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a0c876640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0a080a5100 0x7f0a081350b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:53.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a0c876640 10 monclient(hunting): picked mon.noname-c con 0x7f0a080a5100 addr v2:172.21.15.71:3300/0 2024-04-19T01:48:53.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a0c876640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a081355f0 0x7f0a08135a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:53.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a0c876640 10 monclient(hunting): picked mon.noname-b con 0x7f0a081355f0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:53.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a0c876640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:53.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a0c876640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f0a0809bf00 con 0x7f0a080a47e0 2024-04-19T01:48:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a0c876640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a0c876640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f0a0809ee90 con 0x7f0a080a5100 2024-04-19T01:48:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a0c876640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a0c876640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f0a0809f7c0 con 0x7f0a081355f0 2024-04-19T01:48:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a0c876640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a0c876640 10 monclient(hunting): authenticate will time out at 1319.250020s 2024-04-19T01:48:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a06d9d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a081355f0 0x7f0a08135a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a0659c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0a080a47e0 0x7f0a080a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a06d9d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a081355f0 0x7f0a08135a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:58168/0 (socket says 172.21.15.62:58168) 2024-04-19T01:48:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a06d9d640 1 -- 172.21.15.62:0/432168633 learned_addr learned my addr 172.21.15.62:0/432168633 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a06d9d640 10 monclient(hunting): get_auth_request con 0x7f0a081355f0 auth_method 0 2024-04-19T01:48:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a06d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a06d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a06d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:53.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a0659c640 10 monclient(hunting): get_auth_request con 0x7f0a080a47e0 auth_method 0 2024-04-19T01:48:53.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a0659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:53.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a0659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:53.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a0659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:53.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a06d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:53.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a06d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:53.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a06d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:53.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a0659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:53.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a0659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:53.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a0659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:53.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a05d9b640 1 --2- 172.21.15.62:0/432168633 >> v2:172.21.15.71:3300/0 conn(0x7f0a080a5100 0x7f0a081350b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:53.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a05d9b640 10 monclient(hunting): get_auth_request con 0x7f0a080a5100 auth_method 0 2024-04-19T01:48:53.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a05d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:53.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a05d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:53.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a05d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:53.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a06d9d640 10 monclient(hunting): handle_auth_done global_id 34118 payload 274 2024-04-19T01:48:53.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a06d9d640 1 -- 172.21.15.62:0/432168633 >> v2:172.21.15.71:3300/0 conn(0x7f0a080a5100 msgr2=0x7f0a081350b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:53.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a06d9d640 1 --2- 172.21.15.62:0/432168633 >> v2:172.21.15.71:3300/0 conn(0x7f0a080a5100 0x7f0a081350b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:53.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a06d9d640 1 -- 172.21.15.62:0/432168633 >> v2:172.21.15.62:3300/0 conn(0x7f0a080a47e0 msgr2=0x7f0a080a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:53.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a06d9d640 1 --2- 172.21.15.62:0/432168633 >> v2:172.21.15.62:3300/0 conn(0x7f0a080a47e0 0x7f0a080a4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:53.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a06d9d640 10 monclient: _finish_hunting 0 2024-04-19T01:48:53.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a06d9d640 1 monclient: found mon.noname-b 2024-04-19T01:48:53.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a06d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:53.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a06d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:48:53.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a06d9d640 1 -- 172.21.15.62:0/432168633 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a0809c2d0 con 0x7f0a081355f0 2024-04-19T01:48:53.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.429+0000 7f0a0659c640 1 --2- 172.21.15.62:0/432168633 >> v2:172.21.15.62:3300/0 conn(0x7f0a080a47e0 0x7f0a080a4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:48:53.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.433+0000 7f0a06d9d640 1 -- 172.21.15.62:0/432168633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a081355f0 msgr2=0x7f0a08135a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:53.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.433+0000 7f0a06d9d640 1 --2- 172.21.15.62:0/432168633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a081355f0 0x7f0a08135a80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0a00009e80 tx=0x7f0a0001c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11ae1fb0b68cce4e server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:53.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.433+0000 7f0a0559a640 1 -- 172.21.15.62:0/432168633 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a000a5790 con 0x7f0a081355f0 2024-04-19T01:48:53.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.433+0000 7f0a0559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:53.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.433+0000 7f0a0559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:48:53.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.433+0000 7f0a0559a640 10 monclient: dump: 2024-04-19T01:48:53.437 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:53.437 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:53.437 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:53.437 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:53.437 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:53.437 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:53.437 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:53.437 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:53.438 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:53.438 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:53.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.433+0000 7f0a0559a640 10 monclient: _finish_auth 0 2024-04-19T01:48:53.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.433+0000 7f0a0559a640 10 monclient: _check_auth_tickets 2024-04-19T01:48:53.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.433+0000 7f0a0559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:53.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.433+0000 7f0a0c876640 5 monclient: authenticate success, global_id 34118 2024-04-19T01:48:53.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.433+0000 7f0a0c876640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:48:53.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.433+0000 7f0a0559a640 1 -- 172.21.15.62:0/432168633 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a000a5930 con 0x7f0a081355f0 2024-04-19T01:48:53.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.433+0000 7f0a0559a640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:53.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.433+0000 7f0a0c876640 10 monclient: get_monmap_and_config success 2024-04-19T01:48:53.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.433+0000 7f0a0559a640 1 -- 172.21.15.62:0/432168633 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a000a5d00 con 0x7f0a081355f0 2024-04-19T01:48:53.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.433+0000 7f0a0559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:53.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.433+0000 7f0a0c876640 4 set_mon_vals no callback set 2024-04-19T01:48:53.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.433+0000 7f0a0559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:48:53.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.433+0000 7f0a0559a640 10 monclient: dump: 2024-04-19T01:48:53.439 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:53.439 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:53.439 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:53.439 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:53.439 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:53.439 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:53.439 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:53.439 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:53.439 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:53.439 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:53.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.433+0000 7f0a0c876640 10 monclient: shutdown 2024-04-19T01:48:53.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.433+0000 7f0a0c876640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:53.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.433+0000 7f0a0c876640 1 -- 172.21.15.62:0/432168633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a081355f0 msgr2=0x7f0a08135a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:53.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.433+0000 7f0a0c876640 1 --2- 172.21.15.62:0/432168633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a081355f0 0x7f0a08135a80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0a00009e80 tx=0x7f0a0001c8f0 comp rx=0 tx=0).stop 2024-04-19T01:48:53.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.433+0000 7f0a0c876640 1 -- 172.21.15.62:0/432168633 shutdown_connections 2024-04-19T01:48:53.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.433+0000 7f0a0c876640 1 --2- 172.21.15.62:0/432168633 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a081355f0 0x7f0a08135a80 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:53.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.433+0000 7f0a0c876640 1 --2- 172.21.15.62:0/432168633 >> v2:172.21.15.71:3300/0 conn(0x7f0a080a5100 0x7f0a081350b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:53.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.433+0000 7f0a0c876640 1 --2- 172.21.15.62:0/432168633 >> v2:172.21.15.62:3300/0 conn(0x7f0a080a47e0 0x7f0a080a4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:53.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.433+0000 7f0a0c876640 1 -- 172.21.15.62:0/432168633 >> 172.21.15.62:0/432168633 conn(0x7f0a080a23d0 msgr2=0x7f0a080a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:53.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.433+0000 7f0a0c876640 1 -- 172.21.15.62:0/432168633 shutdown_connections 2024-04-19T01:48:53.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.433+0000 7f0a0c876640 1 -- 172.21.15.62:0/432168633 wait complete. 2024-04-19T01:48:53.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a0c876640 10 monclient: build_initial_monmap 2024-04-19T01:48:53.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a0c876640 10 monclient: monmap: 2024-04-19T01:48:53.440 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:53.440 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:48:53.440 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:48:53.440 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:48:53.440 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:53.440 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:53.440 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:48:53.440 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:48:53.441 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:48:53.441 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:53.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a0c876640 1 Processor -- start 2024-04-19T01:48:53.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a0c876640 1 -- start start 2024-04-19T01:48:53.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a0c876640 10 monclient: init 2024-04-19T01:48:53.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a0c876640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:53.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a0c876640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-19T01:48:53.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a0c876640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a080a47e0 0x7f0a0811c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:53.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a0c876640 10 monclient(hunting): picked mon.noname-b con 0x7f0a080a47e0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:48:53.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a0c876640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a080a5100 0x7f0a0813a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:53.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a0c876640 10 monclient(hunting): picked mon.noname-c con 0x7f0a080a5100 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:53.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a0c876640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a081355f0 0x7f0a08132130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:53.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a0c876640 10 monclient(hunting): picked mon.noname-a con 0x7f0a081355f0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:48:53.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a0c876640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:53.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a0c876640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0809f1d0 con 0x7f0a081355f0 2024-04-19T01:48:53.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a0c876640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:53.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a0c876640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0809f050 con 0x7f0a080a47e0 2024-04-19T01:48:53.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a0c876640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:53.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a0c876640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f0a0809c760 con 0x7f0a080a5100 2024-04-19T01:48:53.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a0c876640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:53.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a0c876640 10 monclient(hunting): authenticate will time out at 1319.257138s 2024-04-19T01:48:53.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a05d9b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a080a5100 0x7f0a0813a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:53.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a06d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a081355f0 0x7f0a08132130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:53.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a06d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a081355f0 0x7f0a08132130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47802/0 (socket says 172.21.15.62:47802) 2024-04-19T01:48:53.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a06d9d640 1 -- 172.21.15.62:0/3440435368 learned_addr learned my addr 172.21.15.62:0/3440435368 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:53.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a05d9b640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a080a5100 0x7f0a0813a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:58174/0 (socket says 172.21.15.62:58174) 2024-04-19T01:48:53.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a06d9d640 10 monclient(hunting): get_auth_request con 0x7f0a081355f0 auth_method 0 2024-04-19T01:48:53.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a06d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:53.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a06d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:53.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a06d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:53.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a05d9b640 10 monclient(hunting): get_auth_request con 0x7f0a080a5100 auth_method 0 2024-04-19T01:48:53.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a05d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:53.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a05d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:53.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a05d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:53.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a06d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:53.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a06d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:53.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a06d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:53.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a05d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:53.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a05d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:53.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a0659c640 1 --2- 172.21.15.62:0/3440435368 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a080a47e0 0x7f0a0811c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:53.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a05d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:53.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a0659c640 10 monclient(hunting): get_auth_request con 0x7f0a080a47e0 auth_method 0 2024-04-19T01:48:53.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a0659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:53.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a0659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:53.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a0659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:53.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a05d9b640 10 monclient(hunting): handle_auth_done global_id 34121 payload 824 2024-04-19T01:48:53.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a05d9b640 1 -- 172.21.15.62:0/3440435368 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a080a47e0 msgr2=0x7f0a0811c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:53.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a05d9b640 1 --2- 172.21.15.62:0/3440435368 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a080a47e0 0x7f0a0811c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:53.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a05d9b640 1 -- 172.21.15.62:0/3440435368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a081355f0 msgr2=0x7f0a08132130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:53.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a05d9b640 1 --2- 172.21.15.62:0/3440435368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a081355f0 0x7f0a08132130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:53.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a05d9b640 10 monclient: _finish_hunting 0 2024-04-19T01:48:53.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a05d9b640 1 monclient: found mon.noname-c 2024-04-19T01:48:53.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a05d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:53.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a05d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:48:53.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a05d9b640 1 -- 172.21.15.62:0/3440435368 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a0811daf0 con 0x7f0a080a5100 2024-04-19T01:48:53.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a06d9d640 1 --2- 172.21.15.62:0/3440435368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a081355f0 0x7f0a08132130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:48:53.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a05d9b640 1 -- 172.21.15.62:0/3440435368 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a080a5100 msgr2=0x7f0a0813a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:53.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.437+0000 7f0a05d9b640 1 --2- 172.21.15.62:0/3440435368 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a080a5100 0x7f0a0813a2f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f09fc00d620 tx=0x7f09fc00dae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:53.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.441+0000 7f09f6ffd640 1 -- 172.21.15.62:0/3440435368 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09fc00b830 con 0x7f0a080a5100 2024-04-19T01:48:53.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.441+0000 7f09f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:53.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.441+0000 7f09f6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:48:53.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.441+0000 7f09f6ffd640 10 monclient: dump: 2024-04-19T01:48:53.445 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:53.445 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:53.445 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:53.445 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:53.445 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:53.445 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:53.445 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:53.445 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:53.445 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:53.445 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:53.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.441+0000 7f09f6ffd640 10 monclient: _finish_auth 0 2024-04-19T01:48:53.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.441+0000 7f09f6ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:48:53.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.441+0000 7f09f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:53.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.441+0000 7f09f6ffd640 1 -- 172.21.15.62:0/3440435368 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09fc019070 con 0x7f0a080a5100 2024-04-19T01:48:53.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.441+0000 7f0a0c876640 5 monclient: authenticate success, global_id 34121 2024-04-19T01:48:53.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.441+0000 7f0a0c876640 10 monclient: _renew_subs 2024-04-19T01:48:53.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.441+0000 7f0a0c876640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:48:53.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.441+0000 7f0a0c876640 1 -- 172.21.15.62:0/3440435368 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a080717c0 con 0x7f0a080a5100 2024-04-19T01:48:53.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.441+0000 7f09f6ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:53.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.441+0000 7f09f6ffd640 1 -- 172.21.15.62:0/3440435368 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09fc00bc10 con 0x7f0a080a5100 2024-04-19T01:48:53.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.441+0000 7f09f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:53.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.441+0000 7f09f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:48:53.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.441+0000 7f09f6ffd640 10 monclient: dump: 2024-04-19T01:48:53.447 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:53.447 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:53.447 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:53.447 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:53.447 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:53.447 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:53.447 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:53.447 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:53.447 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:53.447 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:53.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.441+0000 7f0a0c876640 10 monclient: _renew_subs 2024-04-19T01:48:53.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.441+0000 7f0a0c876640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:48:53.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.441+0000 7f0a0c876640 1 -- 172.21.15.62:0/3440435368 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a0813ba80 con 0x7f0a080a5100 2024-04-19T01:48:53.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.441+0000 7f0a04d99640 4 set_mon_vals no callback set 2024-04-19T01:48:53.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.441+0000 7f0a0c876640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:53.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.441+0000 7f0a0c876640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:53.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.441+0000 7f0a0c876640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:48:53.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.441+0000 7f0a0c876640 1 -- 172.21.15.62:0/3440435368 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a080692b0 con 0x7f0a080a5100 2024-04-19T01:48:53.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.441+0000 7f0a05d9b640 1 --2- 172.21.15.62:0/3440435368 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a080a5100 0x7f0a0813a2f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f09fc00d620 tx=0x7f09fc00dae0 comp rx=0 tx=0).queue_received will delay after 1.71349e+09 on 0x7f09fc0496f0 osd_map(325..325 src has 115..325) 2024-04-19T01:48:53.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.441+0000 7f09f6ffd640 1 -- 172.21.15.62:0/3440435368 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f09fc01e460 con 0x7f0a080a5100 2024-04-19T01:48:53.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.441+0000 7f09f6ffd640 1 --2- 172.21.15.62:0/3440435368 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f09e8036420 0x7f09e80388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:53.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.441+0000 7f0a0659c640 1 --2- 172.21.15.62:0/3440435368 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f09e8036420 0x7f09e80388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:53.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.441+0000 7f0a0659c640 10 monclient: get_auth_request con 0x7f09e8036420 auth_method 0 2024-04-19T01:48:53.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.445+0000 7f0a0659c640 1 --2- 172.21.15.62:0/3440435368 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f09e8036420 0x7f09e80388e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f09f8009580 tx=0x7f09f8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:53.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:53.445+0000 7f09f6ffd640 1 -- 172.21.15.62:0/3440435368 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(325..325 src has 115..325) ==== 4346+0+0 (secure 0 0 0) 0x7f09fc0496f0 con 0x7f0a080a5100 2024-04-19T01:48:54.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.269+0000 7f09f6ffd640 1 -- 172.21.15.62:0/3440435368 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f09fc00bdb0 con 0x7f0a080a5100 2024-04-19T01:48:54.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.269+0000 7f09f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:54.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.269+0000 7f09f6ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:48:54.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.445+0000 7f0a0c876640 1 -- 172.21.15.62:0/3440435368 --> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0a0806b980 con 0x7f09e8036420 2024-04-19T01:48:54.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.449+0000 7f09f6ffd640 1 -- 172.21.15.62:0/3440435368 <== mgr.4108 v2:172.21.15.71:6824/2133533992 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33848 (secure 0 0 0) 0x7f0a0806b980 con 0x7f09e8036420 2024-04-19T01:48:54.444 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:48:54.444 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-04-19T01:48:54.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.449+0000 7f0a0c876640 1 -- 172.21.15.62:0/3440435368 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f09e8036420 msgr2=0x7f09e80388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:54.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.449+0000 7f0a0c876640 1 --2- 172.21.15.62:0/3440435368 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f09e8036420 0x7f09e80388e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f09f8009580 tx=0x7f09f8008040 comp rx=0 tx=0).stop 2024-04-19T01:48:54.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.449+0000 7f0a0c876640 10 monclient: shutdown 2024-04-19T01:48:54.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.449+0000 7f0a0c876640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:54.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.449+0000 7f0a0c876640 1 -- 172.21.15.62:0/3440435368 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a080a5100 msgr2=0x7f0a0813a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:54.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.449+0000 7f0a0c876640 1 --2- 172.21.15.62:0/3440435368 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a080a5100 0x7f0a0813a2f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f09fc00d620 tx=0x7f09fc00dae0 comp rx=0 tx=0).stop 2024-04-19T01:48:54.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.449+0000 7f0a0c876640 1 -- 172.21.15.62:0/3440435368 shutdown_connections 2024-04-19T01:48:54.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.449+0000 7f0a0c876640 1 --2- 172.21.15.62:0/3440435368 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f09e8036420 0x7f09e80388e0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:54.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.449+0000 7f0a0c876640 1 --2- 172.21.15.62:0/3440435368 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f0a080a5100 0x7f0a0813a2f0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:54.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.449+0000 7f0a0c876640 1 --2- 172.21.15.62:0/3440435368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a081355f0 0x7f0a08132130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:54.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.449+0000 7f0a0c876640 1 --2- 172.21.15.62:0/3440435368 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a080a47e0 0x7f0a0811c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:54.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.449+0000 7f0a0c876640 1 -- 172.21.15.62:0/3440435368 >> 172.21.15.62:0/3440435368 conn(0x7f0a080a23d0 msgr2=0x7f0a080a11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:54.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.449+0000 7f0a0c876640 1 -- 172.21.15.62:0/3440435368 shutdown_connections 2024-04-19T01:48:54.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.449+0000 7f0a0c876640 1 -- 172.21.15.62:0/3440435368 wait complete. 2024-04-19T01:48:54.463 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":517,"stamp":"2024-04-19T01:48:52.816685+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":101,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":193244,"kb_used_data":18632,"kb_used_omap":9,"kb_used_meta":174518,"kb_avail":566037796,"statfs":{"total":579820584960,"available":579622703104,"internally_reserved":0,"allocated":19079168,"data_stored":7581674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":178707104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.201933"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":620,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:47.760817+0000","last_change":"2024-04-19T01:42:24.162228+0000","last_active":"2024-04-19T01:48:47.760817+0000","last_peered":"2024-04-19T01:48:47.760817+0000","last_clean":"2024-04-19T01:48:47.760817+0000","last_became_active":"2024-04-19T01:42:24.161645+0000","last_became_peered":"2024-04-19T01:42:24.161645+0000","last_unstale":"2024-04-19T01:48:47.760817+0000","last_undegraded":"2024-04-19T01:48:47.760817+0000","last_fullsized":"2024-04-19T01:48:47.760817+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T06:14:08.038010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":617,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:47.491704+0000","last_change":"2024-04-19T01:42:24.132804+0000","last_active":"2024-04-19T01:48:47.491704+0000","last_peered":"2024-04-19T01:48:47.491704+0000","last_clean":"2024-04-19T01:48:47.491704+0000","last_became_active":"2024-04-19T01:42:24.131937+0000","last_became_peered":"2024-04-19T01:42:24.131937+0000","last_unstale":"2024-04-19T01:48:47.491704+0000","last_undegraded":"2024-04-19T01:48:47.491704+0000","last_fullsized":"2024-04-19T01:48:47.491704+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T13:40:26.146488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":621,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:47.760787+0000","last_change":"2024-04-19T01:42:24.135590+0000","last_active":"2024-04-19T01:48:47.760787+0000","last_peered":"2024-04-19T01:48:47.760787+0000","last_clean":"2024-04-19T01:48:47.760787+0000","last_became_active":"2024-04-19T01:42:24.134918+0000","last_became_peered":"2024-04-19T01:42:24.134918+0000","last_unstale":"2024-04-19T01:48:47.760787+0000","last_undegraded":"2024-04-19T01:48:47.760787+0000","last_fullsized":"2024-04-19T01:48:47.760787+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T11:52:05.398825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":618,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:47.491620+0000","last_change":"2024-04-19T01:42:24.132575+0000","last_active":"2024-04-19T01:48:47.491620+0000","last_peered":"2024-04-19T01:48:47.491620+0000","last_clean":"2024-04-19T01:48:47.491620+0000","last_became_active":"2024-04-19T01:42:24.131894+0000","last_became_peered":"2024-04-19T01:42:24.131894+0000","last_unstale":"2024-04-19T01:48:47.491620+0000","last_undegraded":"2024-04-19T01:48:47.491620+0000","last_fullsized":"2024-04-19T01:48:47.491620+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T07:32:38.880998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":615,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:46.659613+0000","last_change":"2024-04-19T01:42:24.145699+0000","last_active":"2024-04-19T01:48:46.659613+0000","last_peered":"2024-04-19T01:48:46.659613+0000","last_clean":"2024-04-19T01:48:46.659613+0000","last_became_active":"2024-04-19T01:42:24.144746+0000","last_became_peered":"2024-04-19T01:42:24.144746+0000","last_unstale":"2024-04-19T01:48:46.659613+0000","last_undegraded":"2024-04-19T01:48:46.659613+0000","last_fullsized":"2024-04-19T01:48:46.659613+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:21:11.400175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":617,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:47.762428+0000","last_change":"2024-04-19T01:42:24.142754+0000","last_active":"2024-04-19T01:48:47.762428+0000","last_peered":"2024-04-19T01:48:47.762428+0000","last_clean":"2024-04-19T01:48:47.762428+0000","last_became_active":"2024-04-19T01:42:24.142309+0000","last_became_peered":"2024-04-19T01:42:24.142309+0000","last_unstale":"2024-04-19T01:48:47.762428+0000","last_undegraded":"2024-04-19T01:48:47.762428+0000","last_fullsized":"2024-04-19T01:48:47.762428+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:40:28.826490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":621,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:47.760787+0000","last_change":"2024-04-19T01:42:24.134484+0000","last_active":"2024-04-19T01:48:47.760787+0000","last_peered":"2024-04-19T01:48:47.760787+0000","last_clean":"2024-04-19T01:48:47.760787+0000","last_became_active":"2024-04-19T01:42:24.133282+0000","last_became_peered":"2024-04-19T01:42:24.133282+0000","last_unstale":"2024-04-19T01:48:47.760787+0000","last_undegraded":"2024-04-19T01:48:47.760787+0000","last_fullsized":"2024-04-19T01:48:47.760787+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T13:07:26.373782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":615,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:46.659609+0000","last_change":"2024-04-19T01:42:24.146550+0000","last_active":"2024-04-19T01:48:46.659609+0000","last_peered":"2024-04-19T01:48:46.659609+0000","last_clean":"2024-04-19T01:48:46.659609+0000","last_became_active":"2024-04-19T01:42:24.145667+0000","last_became_peered":"2024-04-19T01:42:24.145667+0000","last_unstale":"2024-04-19T01:48:46.659609+0000","last_undegraded":"2024-04-19T01:48:46.659609+0000","last_fullsized":"2024-04-19T01:48:46.659609+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T10:07:16.240297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":776,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:47.760789+0000","last_change":"2024-04-19T01:42:22.132251+0000","last_active":"2024-04-19T01:48:47.760789+0000","last_peered":"2024-04-19T01:48:47.760789+0000","last_clean":"2024-04-19T01:48:47.760789+0000","last_became_active":"2024-04-19T01:42:22.131432+0000","last_became_peered":"2024-04-19T01:42:22.131432+0000","last_unstale":"2024-04-19T01:48:47.760789+0000","last_undegraded":"2024-04-19T01:48:47.760789+0000","last_fullsized":"2024-04-19T01:48:47.760789+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:21.102542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:21.102542+0000","last_clean_scrub_stamp":"2024-04-19T01:42:21.102542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T06:47:12.046147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":101,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":101,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607633,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34112,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337728,"statfs":{"total":96636764160,"available":96601833472,"internally_reserved":0,"allocated":3002368,"data_stored":1087303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640336,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34112,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337728,"statfs":{"total":96636764160,"available":96601833472,"internally_reserved":0,"allocated":3002368,"data_stored":1087303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640337,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30476,"kb_used_data":3452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341364,"statfs":{"total":96636764160,"available":96605556736,"internally_reserved":0,"allocated":3534848,"data_stored":1616231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640337,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29956,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341884,"statfs":{"total":96636764160,"available":96606089216,"internally_reserved":0,"allocated":3002368,"data_stored":1087303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640337,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29952,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341888,"statfs":{"total":96636764160,"available":96606093312,"internally_reserved":0,"allocated":3002368,"data_stored":1087303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640338,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34636,"kb_used_data":3452,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337204,"statfs":{"total":96636764160,"available":96601296896,"internally_reserved":0,"allocated":3534848,"data_stored":1616231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:48:54.464 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:48:54.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.585+0000 7f529f22f640 10 monclient: get_monmap_and_config 2024-04-19T01:48:54.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.585+0000 7f529f22f640 10 monclient: build_initial_monmap 2024-04-19T01:48:54.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.585+0000 7f529f22f640 10 monclient: monmap: 2024-04-19T01:48:54.583 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:54.583 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:54.583 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:48:54.588676+0000 2024-04-19T01:48:54.583 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:48:54.588676+0000 2024-04-19T01:48:54.583 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:54.583 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:54.583 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:48:54.583 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:48:54.583 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:48:54.583 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:48:54.583 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:48:54.583 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:54.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.585+0000 7f529f22f640 1 Processor -- start 2024-04-19T01:48:54.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.585+0000 7f529f22f640 1 -- start start 2024-04-19T01:48:54.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.585+0000 7f529f22f640 10 monclient: init 2024-04-19T01:48:54.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529f22f640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:54.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529f22f640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-04-19T01:48:54.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529f22f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5298137780 0x7f5298137b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:54.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529f22f640 10 monclient(hunting): picked mon.noname-b con 0x7f5298137780 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:54.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529f22f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f52981380a0 addr v1:172.21.15.71:6789/0 2024-04-19T01:48:54.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529f22f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f52980a60b0 0x7f52980a4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:54.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529f22f640 10 monclient(hunting): picked mon.noname-a con 0x7f52980a60b0 addr v2:172.21.15.62:3300/0 2024-04-19T01:48:54.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529f22f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f529809c2e0 con 0x7f52981380a0 2024-04-19T01:48:54.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529f22f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:54.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529f22f640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f529809bd20 con 0x7f52980a60b0 2024-04-19T01:48:54.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529f22f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:54.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529f22f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f529809c000 con 0x7f5298137780 2024-04-19T01:48:54.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529f22f640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:54.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529f22f640 10 monclient(hunting): authenticate will time out at 1320.408687s 2024-04-19T01:48:54.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529cfcb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5298137780 0x7f5298137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:54.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529d7cc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f52980a60b0 0x7f52980a4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:54.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529cfcb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5298137780 0x7f5298137b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:58184/0 (socket says 172.21.15.62:58184) 2024-04-19T01:48:54.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f5297fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f52981380a0 0x7f52980a45d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.62:40368/0 (socket says 172.21.15.62:40368) 2024-04-19T01:48:54.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f5297fff640 1 -- 172.21.15.62:0/924231471 learned_addr learned my addr 172.21.15.62:0/924231471 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:48:54.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529d7cc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f52980a60b0 0x7f52980a4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47806/0 (socket says 172.21.15.62:47806) 2024-04-19T01:48:54.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529cfcb640 10 monclient(hunting): get_auth_request con 0x7f5298137780 auth_method 0 2024-04-19T01:48:54.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529cfcb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:54.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529cfcb640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:54.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529cfcb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:54.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529d7cc640 10 monclient(hunting): get_auth_request con 0x7f52980a60b0 auth_method 0 2024-04-19T01:48:54.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529d7cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:54.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529d7cc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:54.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529d7cc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:54.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529d7cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:54.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529d7cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:54.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529d7cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:54.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529cfcb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:54.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529cfcb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:54.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529cfcb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:54.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529cfcb640 10 monclient(hunting): handle_auth_done global_id 34124 payload 274 2024-04-19T01:48:54.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529cfcb640 1 -- 172.21.15.62:0/924231471 >> v2:172.21.15.62:3300/0 conn(0x7f52980a60b0 msgr2=0x7f52980a4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:54.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529cfcb640 1 --2- 172.21.15.62:0/924231471 >> v2:172.21.15.62:3300/0 conn(0x7f52980a60b0 0x7f52980a4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:54.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529cfcb640 1 -- 172.21.15.62:0/924231471 >> v1:172.21.15.71:6789/0 conn(0x7f52981380a0 legacy=0x7f52980a45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:54.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529cfcb640 10 monclient: _finish_hunting 0 2024-04-19T01:48:54.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529cfcb640 1 monclient: found mon.noname-b 2024-04-19T01:48:54.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529cfcb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:54.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529cfcb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:48:54.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529cfcb640 1 -- 172.21.15.62:0/924231471 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5298073210 con 0x7f5298137780 2024-04-19T01:48:54.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.589+0000 7f529d7cc640 1 --2- 172.21.15.62:0/924231471 >> v2:172.21.15.62:3300/0 conn(0x7f52980a60b0 0x7f52980a4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:48:54.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f529cfcb640 1 -- 172.21.15.62:0/924231471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5298137780 msgr2=0x7f5298137b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:54.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f529cfcb640 1 --2- 172.21.15.62:0/924231471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5298137780 0x7f5298137b60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f52880097c0 tx=0x7f528801c180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fb8ae60cc9fc5e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:54.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f52977fe640 1 -- 172.21.15.62:0/924231471 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52880b0070 con 0x7f5298137780 2024-04-19T01:48:54.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f52977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:54.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f52977fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:48:54.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f52977fe640 10 monclient: dump: 2024-04-19T01:48:54.594 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:54.594 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:54.594 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:54.594 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:54.594 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:54.594 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:54.594 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:54.594 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:54.594 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:54.594 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:54.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f52977fe640 10 monclient: _finish_auth 0 2024-04-19T01:48:54.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f52977fe640 10 monclient: _check_auth_tickets 2024-04-19T01:48:54.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f52977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:54.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f52977fe640 1 -- 172.21.15.62:0/924231471 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52880ab600 con 0x7f5298137780 2024-04-19T01:48:54.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f52977fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:54.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f52977fe640 1 -- 172.21.15.62:0/924231471 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52880ab9d0 con 0x7f5298137780 2024-04-19T01:48:54.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f529f22f640 5 monclient: authenticate success, global_id 34124 2024-04-19T01:48:54.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f529f22f640 10 monclient: get_monmap_and_config success 2024-04-19T01:48:54.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f529f22f640 4 set_mon_vals no callback set 2024-04-19T01:48:54.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f52977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:54.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f52977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:48:54.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f52977fe640 10 monclient: dump: 2024-04-19T01:48:54.595 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:54.595 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:54.595 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:54.596 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:54.596 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:54.596 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:54.596 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:54.596 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:54.596 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:54.596 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:54.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f529f22f640 10 monclient: shutdown 2024-04-19T01:48:54.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f529f22f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:54.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f529f22f640 1 -- 172.21.15.62:0/924231471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5298137780 msgr2=0x7f5298137b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:54.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f529f22f640 1 --2- 172.21.15.62:0/924231471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5298137780 0x7f5298137b60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f52880097c0 tx=0x7f528801c180 comp rx=0 tx=0).stop 2024-04-19T01:48:54.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f529f22f640 1 -- 172.21.15.62:0/924231471 shutdown_connections 2024-04-19T01:48:54.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f529f22f640 1 --2- 172.21.15.62:0/924231471 >> v2:172.21.15.62:3300/0 conn(0x7f52980a60b0 0x7f52980a4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:54.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f529f22f640 1 --2- 172.21.15.62:0/924231471 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5298137780 0x7f5298137b60 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:54.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f529f22f640 1 -- 172.21.15.62:0/924231471 >> 172.21.15.62:0/924231471 conn(0x7f52980aa1d0 msgr2=0x7f52980a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:54.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f529f22f640 1 -- 172.21.15.62:0/924231471 shutdown_connections 2024-04-19T01:48:54.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f529f22f640 1 -- 172.21.15.62:0/924231471 wait complete. 2024-04-19T01:48:54.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f529f22f640 10 monclient: build_initial_monmap 2024-04-19T01:48:54.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f529f22f640 10 monclient: monmap: 2024-04-19T01:48:54.597 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:54.597 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:48:54.597 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:48:54.597 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:48:54.597 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:54.597 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:54.597 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:48:54.598 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:48:54.598 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:48:54.598 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:54.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f529f22f640 1 Processor -- start 2024-04-19T01:48:54.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f529f22f640 1 -- start start 2024-04-19T01:48:54.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.593+0000 7f529f22f640 10 monclient: init 2024-04-19T01:48:54.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529f22f640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:54.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529f22f640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-19T01:48:54.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529f22f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52980a60b0 0x7f52981cf5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:54.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529f22f640 10 monclient(hunting): picked mon.noname-c con 0x7f52980a60b0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:54.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529f22f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5298137780 0x7f52981364f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:54.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529f22f640 10 monclient(hunting): picked mon.noname-b con 0x7f5298137780 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:48:54.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529f22f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52981380a0 0x7f529813c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:54.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529f22f640 10 monclient(hunting): picked mon.noname-a con 0x7f52981380a0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:48:54.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529f22f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:54.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529f22f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5298142170 con 0x7f52981380a0 2024-04-19T01:48:54.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529f22f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:54.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529f22f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f52980a5430 con 0x7f5298137780 2024-04-19T01:48:54.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529f22f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:54.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529f22f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f52981422f0 con 0x7f52980a60b0 2024-04-19T01:48:54.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529f22f640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:54.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529f22f640 10 monclient(hunting): authenticate will time out at 1320.416202s 2024-04-19T01:48:54.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529cfcb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52980a60b0 0x7f52981cf5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:54.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529cfcb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52980a60b0 0x7f52981cf5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:58200/0 (socket says 172.21.15.62:58200) 2024-04-19T01:48:54.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529cfcb640 1 -- 172.21.15.62:0/1006141009 learned_addr learned my addr 172.21.15.62:0/1006141009 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:54.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529cfcb640 10 monclient(hunting): get_auth_request con 0x7f52980a60b0 auth_method 0 2024-04-19T01:48:54.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529cfcb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:54.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529cfcb640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:54.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529cfcb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:54.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529d7cc640 1 --2- 172.21.15.62:0/1006141009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52981380a0 0x7f529813c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:54.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529d7cc640 10 monclient(hunting): get_auth_request con 0x7f52981380a0 auth_method 0 2024-04-19T01:48:54.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529d7cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:54.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529d7cc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:54.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529d7cc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:54.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529cfcb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:54.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529cfcb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:54.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529cfcb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:54.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529d7cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:54.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529d7cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:54.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529d7cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:54.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f5297fff640 1 --2- 172.21.15.62:0/1006141009 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5298137780 0x7f52981364f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:54.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f5297fff640 10 monclient(hunting): get_auth_request con 0x7f5298137780 auth_method 0 2024-04-19T01:48:54.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f5297fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:54.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f5297fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:54.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f5297fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:54.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529cfcb640 10 monclient(hunting): handle_auth_done global_id 34127 payload 824 2024-04-19T01:48:54.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529cfcb640 1 -- 172.21.15.62:0/1006141009 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5298137780 msgr2=0x7f52981364f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:54.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529cfcb640 1 --2- 172.21.15.62:0/1006141009 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5298137780 0x7f52981364f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:54.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529cfcb640 1 -- 172.21.15.62:0/1006141009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52981380a0 msgr2=0x7f529813c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:54.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529cfcb640 1 --2- 172.21.15.62:0/1006141009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52981380a0 0x7f529813c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:54.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529cfcb640 10 monclient: _finish_hunting 0 2024-04-19T01:48:54.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529cfcb640 1 monclient: found mon.noname-c 2024-04-19T01:48:54.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529cfcb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:54.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529cfcb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:48:54.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529cfcb640 1 -- 172.21.15.62:0/1006141009 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f529813e0b0 con 0x7f52980a60b0 2024-04-19T01:48:54.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529d7cc640 1 --2- 172.21.15.62:0/1006141009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52981380a0 0x7f529813c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:48:54.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529cfcb640 1 -- 172.21.15.62:0/1006141009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52980a60b0 msgr2=0x7f52981cf5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:54.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529cfcb640 1 --2- 172.21.15.62:0/1006141009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52980a60b0 0x7f52981cf5a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f52880097c0 tx=0x7f528801a9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:54.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f52957fa640 1 -- 172.21.15.62:0/1006141009 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52880b0050 con 0x7f52980a60b0 2024-04-19T01:48:54.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f52957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:54.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f52957fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:48:54.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f52957fa640 10 monclient: dump: 2024-04-19T01:48:54.604 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:54.604 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:54.604 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:54.604 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:54.604 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:54.604 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:54.604 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:54.604 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:54.604 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:54.604 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:54.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f52957fa640 10 monclient: _finish_auth 0 2024-04-19T01:48:54.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f52957fa640 10 monclient: _check_auth_tickets 2024-04-19T01:48:54.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f52957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:54.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f52957fa640 1 -- 172.21.15.62:0/1006141009 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52880bc070 con 0x7f52980a60b0 2024-04-19T01:48:54.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f52957fa640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:54.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f52957fa640 1 -- 172.21.15.62:0/1006141009 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52880b5a40 con 0x7f52980a60b0 2024-04-19T01:48:54.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f52957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:54.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f52957fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:48:54.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f52957fa640 10 monclient: dump: 2024-04-19T01:48:54.605 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:54.605 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:54.605 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:54.605 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:54.605 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:54.605 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:54.605 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:54.605 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:54.605 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:54.605 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:54.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529f22f640 5 monclient: authenticate success, global_id 34127 2024-04-19T01:48:54.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529f22f640 10 monclient: _renew_subs 2024-04-19T01:48:54.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f5296ffd640 4 set_mon_vals no callback set 2024-04-19T01:48:54.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529f22f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:48:54.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529f22f640 1 -- 172.21.15.62:0/1006141009 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52980a3420 con 0x7f52980a60b0 2024-04-19T01:48:54.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529f22f640 10 monclient: _renew_subs 2024-04-19T01:48:54.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529f22f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:48:54.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.597+0000 7f529f22f640 1 -- 172.21.15.62:0/1006141009 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5298120e80 con 0x7f52980a60b0 2024-04-19T01:48:54.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.601+0000 7f52957fa640 1 -- 172.21.15.62:0/1006141009 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f52880b5c60 con 0x7f52980a60b0 2024-04-19T01:48:54.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.601+0000 7f529f22f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:54.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.601+0000 7f529f22f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:54.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.601+0000 7f529f22f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:48:54.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.601+0000 7f529f22f640 1 -- 172.21.15.62:0/1006141009 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5298068ab0 con 0x7f52980a60b0 2024-04-19T01:48:54.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.601+0000 7f52957fa640 1 --2- 172.21.15.62:0/1006141009 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5278036470 0x7f5278038930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:54.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.601+0000 7f52957fa640 1 -- 172.21.15.62:0/1006141009 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(325..325 src has 115..325) ==== 4346+0+0 (secure 0 0 0) 0x7f52880e3990 con 0x7f52980a60b0 2024-04-19T01:48:54.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.601+0000 7f5297fff640 1 --2- 172.21.15.62:0/1006141009 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5278036470 0x7f5278038930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:54.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.601+0000 7f5297fff640 10 monclient: get_auth_request con 0x7f5278036470 auth_method 0 2024-04-19T01:48:54.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.601+0000 7f5297fff640 1 --2- 172.21.15.62:0/1006141009 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5278036470 0x7f5278038930 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5280007fd0 tx=0x7f5280009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:54.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.605+0000 7f52957fa640 1 -- 172.21.15.62:0/1006141009 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f529809c2e0 con 0x7f52980a60b0 2024-04-19T01:48:54.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.605+0000 7f52957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:54.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.605+0000 7f52957fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:48:54.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.761+0000 7f529f22f640 1 -- 172.21.15.62:0/1006141009 --> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5298120a50 con 0x7f5278036470 2024-04-19T01:48:54.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.765+0000 7f52957fa640 1 -- 172.21.15.62:0/1006141009 <== mgr.4108 v2:172.21.15.71:6824/2133533992 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33848 (secure 0 0 0) 0x7f5280020450 con 0x7f5278036470 2024-04-19T01:48:54.763 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:48:54.763 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-04-19T01:48:54.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.769+0000 7f529f22f640 1 -- 172.21.15.62:0/1006141009 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5278036470 msgr2=0x7f5278038930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:54.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.769+0000 7f529f22f640 1 --2- 172.21.15.62:0/1006141009 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5278036470 0x7f5278038930 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5280007fd0 tx=0x7f5280009040 comp rx=0 tx=0).stop 2024-04-19T01:48:54.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.769+0000 7f529f22f640 10 monclient: shutdown 2024-04-19T01:48:54.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.769+0000 7f529f22f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:54.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.769+0000 7f529f22f640 1 -- 172.21.15.62:0/1006141009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52980a60b0 msgr2=0x7f52981cf5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:54.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.769+0000 7f529f22f640 1 --2- 172.21.15.62:0/1006141009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52980a60b0 0x7f52981cf5a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f52880097c0 tx=0x7f528801a9c0 comp rx=0 tx=0).stop 2024-04-19T01:48:54.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.769+0000 7f529f22f640 1 -- 172.21.15.62:0/1006141009 shutdown_connections 2024-04-19T01:48:54.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.769+0000 7f529f22f640 1 --2- 172.21.15.62:0/1006141009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52981380a0 0x7f529813c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:54.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.769+0000 7f529f22f640 1 --2- 172.21.15.62:0/1006141009 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5298137780 0x7f52981364f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:54.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.769+0000 7f529f22f640 1 --2- 172.21.15.62:0/1006141009 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5278036470 0x7f5278038930 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:54.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.769+0000 7f529f22f640 1 --2- 172.21.15.62:0/1006141009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f52980a60b0 0x7f52981cf5a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:54.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.769+0000 7f529f22f640 1 -- 172.21.15.62:0/1006141009 >> 172.21.15.62:0/1006141009 conn(0x7f52980aa1d0 msgr2=0x7f52980a79d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:54.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.769+0000 7f529f22f640 1 -- 172.21.15.62:0/1006141009 shutdown_connections 2024-04-19T01:48:54.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.769+0000 7f529f22f640 1 -- 172.21.15.62:0/1006141009 wait complete. 2024-04-19T01:48:54.780 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":517,"stamp":"2024-04-19T01:48:52.816685+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":101,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":193244,"kb_used_data":18632,"kb_used_omap":9,"kb_used_meta":174518,"kb_avail":566037796,"statfs":{"total":579820584960,"available":579622703104,"internally_reserved":0,"allocated":19079168,"data_stored":7581674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":178707104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.201933"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":620,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:47.760817+0000","last_change":"2024-04-19T01:42:24.162228+0000","last_active":"2024-04-19T01:48:47.760817+0000","last_peered":"2024-04-19T01:48:47.760817+0000","last_clean":"2024-04-19T01:48:47.760817+0000","last_became_active":"2024-04-19T01:42:24.161645+0000","last_became_peered":"2024-04-19T01:42:24.161645+0000","last_unstale":"2024-04-19T01:48:47.760817+0000","last_undegraded":"2024-04-19T01:48:47.760817+0000","last_fullsized":"2024-04-19T01:48:47.760817+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T06:14:08.038010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":617,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:47.491704+0000","last_change":"2024-04-19T01:42:24.132804+0000","last_active":"2024-04-19T01:48:47.491704+0000","last_peered":"2024-04-19T01:48:47.491704+0000","last_clean":"2024-04-19T01:48:47.491704+0000","last_became_active":"2024-04-19T01:42:24.131937+0000","last_became_peered":"2024-04-19T01:42:24.131937+0000","last_unstale":"2024-04-19T01:48:47.491704+0000","last_undegraded":"2024-04-19T01:48:47.491704+0000","last_fullsized":"2024-04-19T01:48:47.491704+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T13:40:26.146488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":621,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:47.760787+0000","last_change":"2024-04-19T01:42:24.135590+0000","last_active":"2024-04-19T01:48:47.760787+0000","last_peered":"2024-04-19T01:48:47.760787+0000","last_clean":"2024-04-19T01:48:47.760787+0000","last_became_active":"2024-04-19T01:42:24.134918+0000","last_became_peered":"2024-04-19T01:42:24.134918+0000","last_unstale":"2024-04-19T01:48:47.760787+0000","last_undegraded":"2024-04-19T01:48:47.760787+0000","last_fullsized":"2024-04-19T01:48:47.760787+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T11:52:05.398825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":618,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:47.491620+0000","last_change":"2024-04-19T01:42:24.132575+0000","last_active":"2024-04-19T01:48:47.491620+0000","last_peered":"2024-04-19T01:48:47.491620+0000","last_clean":"2024-04-19T01:48:47.491620+0000","last_became_active":"2024-04-19T01:42:24.131894+0000","last_became_peered":"2024-04-19T01:42:24.131894+0000","last_unstale":"2024-04-19T01:48:47.491620+0000","last_undegraded":"2024-04-19T01:48:47.491620+0000","last_fullsized":"2024-04-19T01:48:47.491620+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T07:32:38.880998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":615,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:46.659613+0000","last_change":"2024-04-19T01:42:24.145699+0000","last_active":"2024-04-19T01:48:46.659613+0000","last_peered":"2024-04-19T01:48:46.659613+0000","last_clean":"2024-04-19T01:48:46.659613+0000","last_became_active":"2024-04-19T01:42:24.144746+0000","last_became_peered":"2024-04-19T01:42:24.144746+0000","last_unstale":"2024-04-19T01:48:46.659613+0000","last_undegraded":"2024-04-19T01:48:46.659613+0000","last_fullsized":"2024-04-19T01:48:46.659613+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:21:11.400175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":617,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:47.762428+0000","last_change":"2024-04-19T01:42:24.142754+0000","last_active":"2024-04-19T01:48:47.762428+0000","last_peered":"2024-04-19T01:48:47.762428+0000","last_clean":"2024-04-19T01:48:47.762428+0000","last_became_active":"2024-04-19T01:42:24.142309+0000","last_became_peered":"2024-04-19T01:42:24.142309+0000","last_unstale":"2024-04-19T01:48:47.762428+0000","last_undegraded":"2024-04-19T01:48:47.762428+0000","last_fullsized":"2024-04-19T01:48:47.762428+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:40:28.826490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":621,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:47.760787+0000","last_change":"2024-04-19T01:42:24.134484+0000","last_active":"2024-04-19T01:48:47.760787+0000","last_peered":"2024-04-19T01:48:47.760787+0000","last_clean":"2024-04-19T01:48:47.760787+0000","last_became_active":"2024-04-19T01:42:24.133282+0000","last_became_peered":"2024-04-19T01:42:24.133282+0000","last_unstale":"2024-04-19T01:48:47.760787+0000","last_undegraded":"2024-04-19T01:48:47.760787+0000","last_fullsized":"2024-04-19T01:48:47.760787+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T13:07:26.373782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":615,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:46.659609+0000","last_change":"2024-04-19T01:42:24.146550+0000","last_active":"2024-04-19T01:48:46.659609+0000","last_peered":"2024-04-19T01:48:46.659609+0000","last_clean":"2024-04-19T01:48:46.659609+0000","last_became_active":"2024-04-19T01:42:24.145667+0000","last_became_peered":"2024-04-19T01:42:24.145667+0000","last_unstale":"2024-04-19T01:48:46.659609+0000","last_undegraded":"2024-04-19T01:48:46.659609+0000","last_fullsized":"2024-04-19T01:48:46.659609+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T10:07:16.240297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":776,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:47.760789+0000","last_change":"2024-04-19T01:42:22.132251+0000","last_active":"2024-04-19T01:48:47.760789+0000","last_peered":"2024-04-19T01:48:47.760789+0000","last_clean":"2024-04-19T01:48:47.760789+0000","last_became_active":"2024-04-19T01:42:22.131432+0000","last_became_peered":"2024-04-19T01:42:22.131432+0000","last_unstale":"2024-04-19T01:48:47.760789+0000","last_undegraded":"2024-04-19T01:48:47.760789+0000","last_fullsized":"2024-04-19T01:48:47.760789+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:21.102542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:21.102542+0000","last_clean_scrub_stamp":"2024-04-19T01:42:21.102542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T06:47:12.046147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":101,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":101,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607633,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34112,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337728,"statfs":{"total":96636764160,"available":96601833472,"internally_reserved":0,"allocated":3002368,"data_stored":1087303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640336,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34112,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337728,"statfs":{"total":96636764160,"available":96601833472,"internally_reserved":0,"allocated":3002368,"data_stored":1087303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640337,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30476,"kb_used_data":3452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341364,"statfs":{"total":96636764160,"available":96605556736,"internally_reserved":0,"allocated":3534848,"data_stored":1616231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640337,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29956,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341884,"statfs":{"total":96636764160,"available":96606089216,"internally_reserved":0,"allocated":3002368,"data_stored":1087303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640337,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29952,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341888,"statfs":{"total":96636764160,"available":96606093312,"internally_reserved":0,"allocated":3002368,"data_stored":1087303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640338,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34636,"kb_used_data":3452,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337204,"statfs":{"total":96636764160,"available":96601296896,"internally_reserved":0,"allocated":3534848,"data_stored":1616231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:48:54.781 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-19T01:48:54.781 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:48:54.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.897+0000 7fb9e26b2640 10 monclient: get_monmap_and_config 2024-04-19T01:48:54.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.897+0000 7fb9e26b2640 10 monclient: build_initial_monmap 2024-04-19T01:48:54.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.897+0000 7fb9e26b2640 10 monclient: monmap: 2024-04-19T01:48:54.895 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:54.895 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:54.895 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:48:54.901492+0000 2024-04-19T01:48:54.895 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:48:54.901492+0000 2024-04-19T01:48:54.895 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:54.896 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:54.896 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:48:54.896 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:48:54.896 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:48:54.896 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:48:54.896 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:48:54.896 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:54.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e26b2640 1 Processor -- start 2024-04-19T01:48:54.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e26b2640 1 -- start start 2024-04-19T01:48:54.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e26b2640 10 monclient: init 2024-04-19T01:48:54.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e26b2640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:54.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e26b2640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-04-19T01:48:54.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e26b2640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb9dc134660 0x7fb9dc134a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:54.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e26b2640 10 monclient(hunting): picked mon.noname-a con 0x7fb9dc134660 addr v2:172.21.15.62:3300/0 2024-04-19T01:48:54.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e26b2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb9dc072920 addr v1:172.21.15.62:6789/0 2024-04-19T01:48:54.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e26b2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb9dc0a2b70 0x7fb9dc13d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:54.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e26b2640 10 monclient(hunting): picked mon.noname-b con 0x7fb9dc0a2b70 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:54.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e26b2640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9dc0a0f10 con 0x7fb9dc072920 2024-04-19T01:48:54.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e26b2640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:54.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e26b2640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fb9dc071680 con 0x7fb9dc134660 2024-04-19T01:48:54.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e26b2640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:54.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e26b2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fb9dc071960 con 0x7fb9dc0a2b70 2024-04-19T01:48:54.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e26b2640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:54.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e26b2640 10 monclient(hunting): authenticate will time out at 1320.720960s 2024-04-19T01:48:54.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9dbfff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb9dc134660 0x7fb9dc134a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:54.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e0c4f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb9dc0a2b70 0x7fb9dc13d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:54.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9dbfff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb9dc134660 0x7fb9dc134a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47834/0 (socket says 172.21.15.62:47834) 2024-04-19T01:48:54.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9d3fff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fb9dc072920 0x7fb9dc0a2460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:56730/0 (socket says 172.21.15.62:56730) 2024-04-19T01:48:54.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9dbfff640 1 -- 172.21.15.62:0/3664688415 learned_addr learned my addr 172.21.15.62:0/3664688415 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:54.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9dbfff640 10 monclient(hunting): get_auth_request con 0x7fb9dc134660 auth_method 0 2024-04-19T01:48:54.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:54.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9dbfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:54.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:54.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e0c4f640 10 monclient(hunting): get_auth_request con 0x7fb9dc0a2b70 auth_method 0 2024-04-19T01:48:54.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e0c4f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:54.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e0c4f640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:54.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e0c4f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:54.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e0c4f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:54.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e0c4f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:54.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e0c4f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:54.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:54.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:54.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:54.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e0c4f640 10 monclient(hunting): handle_auth_done global_id 34130 payload 274 2024-04-19T01:48:54.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e0c4f640 1 -- 172.21.15.62:0/3664688415 >> v2:172.21.15.62:3300/0 conn(0x7fb9dc134660 msgr2=0x7fb9dc134a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:54.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e0c4f640 1 --2- 172.21.15.62:0/3664688415 >> v2:172.21.15.62:3300/0 conn(0x7fb9dc134660 0x7fb9dc134a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:54.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e0c4f640 1 -- 172.21.15.62:0/3664688415 >> v1:172.21.15.62:6789/0 conn(0x7fb9dc072920 legacy=0x7fb9dc0a2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:54.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e0c4f640 10 monclient: _finish_hunting 0 2024-04-19T01:48:54.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e0c4f640 1 monclient: found mon.noname-b 2024-04-19T01:48:54.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e0c4f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:54.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e0c4f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:48:54.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9e0c4f640 1 -- 172.21.15.62:0/3664688415 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9dc071c40 con 0x7fb9dc0a2b70 2024-04-19T01:48:54.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.901+0000 7fb9dbfff640 1 --2- 172.21.15.62:0/3664688415 >> v2:172.21.15.62:3300/0 conn(0x7fb9dc134660 0x7fb9dc134a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:48:54.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e0c4f640 1 -- 172.21.15.62:0/3664688415 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb9dc0a2b70 msgr2=0x7fb9dc13d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:54.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e0c4f640 1 --2- 172.21.15.62:0/3664688415 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb9dc0a2b70 0x7fb9dc13d640 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4009d90 tx=0x7fb9d401c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1ec2c51c8f6d1b server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:54.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9db7fe640 1 -- 172.21.15.62:0/3664688415 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d40b2070 con 0x7fb9dc0a2b70 2024-04-19T01:48:54.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:54.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9db7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:48:54.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9db7fe640 10 monclient: dump: 2024-04-19T01:48:54.908 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:54.908 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:54.908 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:54.908 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:54.908 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:54.908 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:54.909 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:54.909 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:54.909 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:54.909 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:54.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9db7fe640 10 monclient: _finish_auth 0 2024-04-19T01:48:54.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9db7fe640 10 monclient: _check_auth_tickets 2024-04-19T01:48:54.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:54.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9db7fe640 1 -- 172.21.15.62:0/3664688415 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9d40a5790 con 0x7fb9dc0a2b70 2024-04-19T01:48:54.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 5 monclient: authenticate success, global_id 34130 2024-04-19T01:48:54.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:48:54.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9db7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:54.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 10 monclient: get_monmap_and_config success 2024-04-19T01:48:54.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9db7fe640 1 -- 172.21.15.62:0/3664688415 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d40a5b60 con 0x7fb9dc0a2b70 2024-04-19T01:48:54.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 4 set_mon_vals no callback set 2024-04-19T01:48:54.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:54.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9db7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:48:54.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9db7fe640 10 monclient: dump: 2024-04-19T01:48:54.910 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:54.910 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:54.910 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:54.910 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:54.910 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:54.910 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:54.910 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:54.910 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:54.910 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:54.910 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:54.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 10 monclient: shutdown 2024-04-19T01:48:54.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:54.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 1 -- 172.21.15.62:0/3664688415 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb9dc0a2b70 msgr2=0x7fb9dc13d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:54.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 1 --2- 172.21.15.62:0/3664688415 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb9dc0a2b70 0x7fb9dc13d640 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4009d90 tx=0x7fb9d401c830 comp rx=0 tx=0).stop 2024-04-19T01:48:54.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 1 -- 172.21.15.62:0/3664688415 shutdown_connections 2024-04-19T01:48:54.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 1 --2- 172.21.15.62:0/3664688415 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb9dc0a2b70 0x7fb9dc13d640 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:54.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 1 --2- 172.21.15.62:0/3664688415 >> v2:172.21.15.62:3300/0 conn(0x7fb9dc134660 0x7fb9dc134a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:54.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 1 -- 172.21.15.62:0/3664688415 >> 172.21.15.62:0/3664688415 conn(0x7fb9dc0a7050 msgr2=0x7fb9dc0a3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:54.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 1 -- 172.21.15.62:0/3664688415 shutdown_connections 2024-04-19T01:48:54.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 1 -- 172.21.15.62:0/3664688415 wait complete. 2024-04-19T01:48:54.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 10 monclient: build_initial_monmap 2024-04-19T01:48:54.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 10 monclient: monmap: 2024-04-19T01:48:54.911 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:54.911 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:48:54.911 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:48:54.911 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:48:54.911 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:54.912 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:54.912 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:48:54.912 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:48:54.912 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:48:54.912 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:54.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 1 Processor -- start 2024-04-19T01:48:54.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 1 -- start start 2024-04-19T01:48:54.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 10 monclient: init 2024-04-19T01:48:54.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:54.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-19T01:48:54.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb9dc072920 0x7fb9dc1cd750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:54.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 10 monclient(hunting): picked mon.noname-c con 0x7fb9dc072920 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:54.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9dc0a2b70 0x7fb9dc133520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:54.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 10 monclient(hunting): picked mon.noname-a con 0x7fb9dc0a2b70 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:48:54.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9dc134660 0x7fb9dc13a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:54.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 10 monclient(hunting): picked mon.noname-b con 0x7fb9dc134660 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:48:54.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:54.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb9dc0a4040 con 0x7fb9dc0a2b70 2024-04-19T01:48:54.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:54.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb9dc0a3ec0 con 0x7fb9dc134660 2024-04-19T01:48:54.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:54.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fb9dc1403a0 con 0x7fb9dc072920 2024-04-19T01:48:54.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:54.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.905+0000 7fb9e26b2640 10 monclient(hunting): authenticate will time out at 1320.727995s 2024-04-19T01:48:54.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9dbfff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb9dc072920 0x7fb9dc1cd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:54.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9dc0a2b70 0x7fb9dc133520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:54.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9dbfff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb9dc072920 0x7fb9dc1cd750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:58220/0 (socket says 172.21.15.62:58220) 2024-04-19T01:48:54.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9dbfff640 1 -- 172.21.15.62:0/3238961009 learned_addr learned my addr 172.21.15.62:0/3238961009 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:54.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9dbfff640 10 monclient(hunting): get_auth_request con 0x7fb9dc072920 auth_method 0 2024-04-19T01:48:54.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:54.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9dbfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:54.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:54.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d3fff640 10 monclient(hunting): get_auth_request con 0x7fb9dc0a2b70 auth_method 0 2024-04-19T01:48:54.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:54.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:54.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:54.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:54.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:54.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:54.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:54.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:54.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:54.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9e0c4f640 1 --2- 172.21.15.62:0/3238961009 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9dc134660 0x7fb9dc13a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:54.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9e0c4f640 10 monclient(hunting): get_auth_request con 0x7fb9dc134660 auth_method 0 2024-04-19T01:48:54.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9e0c4f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:54.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9e0c4f640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:54.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9e0c4f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:54.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d3fff640 10 monclient(hunting): handle_auth_done global_id 15300 payload 824 2024-04-19T01:48:54.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d3fff640 1 -- 172.21.15.62:0/3238961009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb9dc072920 msgr2=0x7fb9dc1cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:54.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d3fff640 1 --2- 172.21.15.62:0/3238961009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb9dc072920 0x7fb9dc1cd750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:54.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d3fff640 1 -- 172.21.15.62:0/3238961009 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9dc134660 msgr2=0x7fb9dc13a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:54.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d3fff640 1 --2- 172.21.15.62:0/3238961009 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9dc134660 0x7fb9dc13a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:54.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d3fff640 10 monclient: _finish_hunting 0 2024-04-19T01:48:54.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d3fff640 1 monclient: found mon.noname-a 2024-04-19T01:48:54.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:54.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:48:54.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d3fff640 1 -- 172.21.15.62:0/3238961009 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9dc0a67d0 con 0x7fb9dc0a2b70 2024-04-19T01:48:54.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9dbfff640 1 --2- 172.21.15.62:0/3238961009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb9dc072920 0x7fb9dc1cd750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:48:54.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d3fff640 1 -- 172.21.15.62:0/3238961009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9dc0a2b70 msgr2=0x7fb9dc133520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:54.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d3fff640 1 --2- 172.21.15.62:0/3238961009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9dc0a2b70 0x7fb9dc133520 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb9c400c3e0 tx=0x7fb9c4005790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:54.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d97fa640 1 -- 172.21.15.62:0/3238961009 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c4017070 con 0x7fb9dc0a2b70 2024-04-19T01:48:54.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:54.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:48:54.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d97fa640 10 monclient: dump: 2024-04-19T01:48:54.916 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:54.916 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:54.916 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:54.916 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:54.916 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:54.916 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:54.916 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:54.916 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:54.916 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:54.916 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:54.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d97fa640 10 monclient: _finish_auth 0 2024-04-19T01:48:54.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d97fa640 10 monclient: _check_auth_tickets 2024-04-19T01:48:54.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:54.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d97fa640 1 -- 172.21.15.62:0/3238961009 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9c4005820 con 0x7fb9dc0a2b70 2024-04-19T01:48:54.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d97fa640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:54.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d97fa640 1 -- 172.21.15.62:0/3238961009 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c4005c10 con 0x7fb9dc0a2b70 2024-04-19T01:48:54.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:54.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:48:54.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d97fa640 10 monclient: dump: 2024-04-19T01:48:54.917 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:54.917 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:54.917 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:54.917 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:54.917 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:54.917 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:54.917 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:54.918 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:54.918 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:54.918 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:54.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9e26b2640 5 monclient: authenticate success, global_id 15300 2024-04-19T01:48:54.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9e26b2640 10 monclient: _renew_subs 2024-04-19T01:48:54.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9e26b2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:54.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9daffd640 4 set_mon_vals no callback set 2024-04-19T01:48:54.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9e26b2640 1 -- 172.21.15.62:0/3238961009 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9dc1ce640 con 0x7fb9dc0a2b70 2024-04-19T01:48:54.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9e26b2640 10 monclient: _renew_subs 2024-04-19T01:48:54.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9e26b2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:54.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9e26b2640 1 -- 172.21.15.62:0/3238961009 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9dc11deb0 con 0x7fb9dc0a2b70 2024-04-19T01:48:54.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.909+0000 7fb9d97fa640 1 -- 172.21.15.62:0/3238961009 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fb9c401a070 con 0x7fb9dc0a2b70 2024-04-19T01:48:54.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.913+0000 7fb9e26b2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:54.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.913+0000 7fb9e26b2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:54.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.913+0000 7fb9e26b2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:54.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.913+0000 7fb9d97fa640 1 --2- 172.21.15.62:0/3238961009 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fb9bc0364a0 0x7fb9bc038960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:54.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.913+0000 7fb9e26b2640 1 -- 172.21.15.62:0/3238961009 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9dc068ab0 con 0x7fb9dc0a2b70 2024-04-19T01:48:54.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.913+0000 7fb9d97fa640 1 -- 172.21.15.62:0/3238961009 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(325..325 src has 115..325) ==== 4346+0+0 (secure 0 0 0) 0x7fb9c4048730 con 0x7fb9dc0a2b70 2024-04-19T01:48:54.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.917+0000 7fb9dbfff640 1 --2- 172.21.15.62:0/3238961009 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fb9bc0364a0 0x7fb9bc038960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:54.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.917+0000 7fb9dbfff640 10 monclient: get_auth_request con 0x7fb9bc0364a0 auth_method 0 2024-04-19T01:48:54.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.917+0000 7fb9d97fa640 1 -- 172.21.15.62:0/3238961009 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb9c40472e0 con 0x7fb9dc0a2b70 2024-04-19T01:48:54.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.917+0000 7fb9d97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:54.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.917+0000 7fb9d97fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:48:54.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:54.917+0000 7fb9dbfff640 1 --2- 172.21.15.62:0/3238961009 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fb9bc0364a0 0x7fb9bc038960 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc13ba30 tx=0x7fb9cc009240 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:55.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.073+0000 7fb9e26b2640 1 -- 172.21.15.62:0/3238961009 --> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb9dc13c710 con 0x7fb9bc0364a0 2024-04-19T01:48:55.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.077+0000 7fb9d97fa640 1 -- 172.21.15.62:0/3238961009 <== mgr.4108 v2:172.21.15.71:6824/2133533992 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33849 (secure 0 0 0) 0x7fb9dc13c710 con 0x7fb9bc0364a0 2024-04-19T01:48:55.074 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:48:55.074 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-04-19T01:48:55.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.077+0000 7fb9e26b2640 1 -- 172.21.15.62:0/3238961009 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fb9bc0364a0 msgr2=0x7fb9bc038960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:55.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.077+0000 7fb9e26b2640 1 --2- 172.21.15.62:0/3238961009 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fb9bc0364a0 0x7fb9bc038960 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc13ba30 tx=0x7fb9cc009240 comp rx=0 tx=0).stop 2024-04-19T01:48:55.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.077+0000 7fb9e26b2640 10 monclient: shutdown 2024-04-19T01:48:55.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.077+0000 7fb9e26b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:55.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.077+0000 7fb9e26b2640 1 -- 172.21.15.62:0/3238961009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9dc0a2b70 msgr2=0x7fb9dc133520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:55.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.077+0000 7fb9e26b2640 1 --2- 172.21.15.62:0/3238961009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9dc0a2b70 0x7fb9dc133520 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb9c400c3e0 tx=0x7fb9c4005790 comp rx=0 tx=0).stop 2024-04-19T01:48:55.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.077+0000 7fb9e26b2640 1 -- 172.21.15.62:0/3238961009 shutdown_connections 2024-04-19T01:48:55.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.077+0000 7fb9e26b2640 1 --2- 172.21.15.62:0/3238961009 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9dc134660 0x7fb9dc13a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:55.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.077+0000 7fb9e26b2640 1 --2- 172.21.15.62:0/3238961009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9dc0a2b70 0x7fb9dc133520 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:55.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.077+0000 7fb9e26b2640 1 --2- 172.21.15.62:0/3238961009 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fb9bc0364a0 0x7fb9bc038960 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:55.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.077+0000 7fb9e26b2640 1 --2- 172.21.15.62:0/3238961009 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb9dc072920 0x7fb9dc1cd750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:55.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.077+0000 7fb9e26b2640 1 -- 172.21.15.62:0/3238961009 >> 172.21.15.62:0/3238961009 conn(0x7fb9dc0a7050 msgr2=0x7fb9dc09be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:55.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.077+0000 7fb9e26b2640 1 -- 172.21.15.62:0/3238961009 shutdown_connections 2024-04-19T01:48:55.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.077+0000 7fb9e26b2640 1 -- 172.21.15.62:0/3238961009 wait complete. 2024-04-19T01:48:55.091 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":518,"stamp":"2024-04-19T01:48:54.817241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":101,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":193244,"kb_used_data":18632,"kb_used_omap":9,"kb_used_meta":174518,"kb_avail":566037796,"statfs":{"total":579820584960,"available":579622703104,"internally_reserved":0,"allocated":19079168,"data_stored":7581674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":178707104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002553"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":620,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:47.760817+0000","last_change":"2024-04-19T01:42:24.162228+0000","last_active":"2024-04-19T01:48:47.760817+0000","last_peered":"2024-04-19T01:48:47.760817+0000","last_clean":"2024-04-19T01:48:47.760817+0000","last_became_active":"2024-04-19T01:42:24.161645+0000","last_became_peered":"2024-04-19T01:42:24.161645+0000","last_unstale":"2024-04-19T01:48:47.760817+0000","last_undegraded":"2024-04-19T01:48:47.760817+0000","last_fullsized":"2024-04-19T01:48:47.760817+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T06:14:08.038010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":617,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:47.491704+0000","last_change":"2024-04-19T01:42:24.132804+0000","last_active":"2024-04-19T01:48:47.491704+0000","last_peered":"2024-04-19T01:48:47.491704+0000","last_clean":"2024-04-19T01:48:47.491704+0000","last_became_active":"2024-04-19T01:42:24.131937+0000","last_became_peered":"2024-04-19T01:42:24.131937+0000","last_unstale":"2024-04-19T01:48:47.491704+0000","last_undegraded":"2024-04-19T01:48:47.491704+0000","last_fullsized":"2024-04-19T01:48:47.491704+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T13:40:26.146488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":621,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:47.760787+0000","last_change":"2024-04-19T01:42:24.135590+0000","last_active":"2024-04-19T01:48:47.760787+0000","last_peered":"2024-04-19T01:48:47.760787+0000","last_clean":"2024-04-19T01:48:47.760787+0000","last_became_active":"2024-04-19T01:42:24.134918+0000","last_became_peered":"2024-04-19T01:42:24.134918+0000","last_unstale":"2024-04-19T01:48:47.760787+0000","last_undegraded":"2024-04-19T01:48:47.760787+0000","last_fullsized":"2024-04-19T01:48:47.760787+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T11:52:05.398825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":618,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:47.491620+0000","last_change":"2024-04-19T01:42:24.132575+0000","last_active":"2024-04-19T01:48:47.491620+0000","last_peered":"2024-04-19T01:48:47.491620+0000","last_clean":"2024-04-19T01:48:47.491620+0000","last_became_active":"2024-04-19T01:42:24.131894+0000","last_became_peered":"2024-04-19T01:42:24.131894+0000","last_unstale":"2024-04-19T01:48:47.491620+0000","last_undegraded":"2024-04-19T01:48:47.491620+0000","last_fullsized":"2024-04-19T01:48:47.491620+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T07:32:38.880998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":615,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:46.659613+0000","last_change":"2024-04-19T01:42:24.145699+0000","last_active":"2024-04-19T01:48:46.659613+0000","last_peered":"2024-04-19T01:48:46.659613+0000","last_clean":"2024-04-19T01:48:46.659613+0000","last_became_active":"2024-04-19T01:42:24.144746+0000","last_became_peered":"2024-04-19T01:42:24.144746+0000","last_unstale":"2024-04-19T01:48:46.659613+0000","last_undegraded":"2024-04-19T01:48:46.659613+0000","last_fullsized":"2024-04-19T01:48:46.659613+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:21:11.400175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":617,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:47.762428+0000","last_change":"2024-04-19T01:42:24.142754+0000","last_active":"2024-04-19T01:48:47.762428+0000","last_peered":"2024-04-19T01:48:47.762428+0000","last_clean":"2024-04-19T01:48:47.762428+0000","last_became_active":"2024-04-19T01:42:24.142309+0000","last_became_peered":"2024-04-19T01:42:24.142309+0000","last_unstale":"2024-04-19T01:48:47.762428+0000","last_undegraded":"2024-04-19T01:48:47.762428+0000","last_fullsized":"2024-04-19T01:48:47.762428+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:40:28.826490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":621,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:47.760787+0000","last_change":"2024-04-19T01:42:24.134484+0000","last_active":"2024-04-19T01:48:47.760787+0000","last_peered":"2024-04-19T01:48:47.760787+0000","last_clean":"2024-04-19T01:48:47.760787+0000","last_became_active":"2024-04-19T01:42:24.133282+0000","last_became_peered":"2024-04-19T01:42:24.133282+0000","last_unstale":"2024-04-19T01:48:47.760787+0000","last_undegraded":"2024-04-19T01:48:47.760787+0000","last_fullsized":"2024-04-19T01:48:47.760787+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T13:07:26.373782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":615,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:46.659609+0000","last_change":"2024-04-19T01:42:24.146550+0000","last_active":"2024-04-19T01:48:46.659609+0000","last_peered":"2024-04-19T01:48:46.659609+0000","last_clean":"2024-04-19T01:48:46.659609+0000","last_became_active":"2024-04-19T01:42:24.145667+0000","last_became_peered":"2024-04-19T01:42:24.145667+0000","last_unstale":"2024-04-19T01:48:46.659609+0000","last_undegraded":"2024-04-19T01:48:46.659609+0000","last_fullsized":"2024-04-19T01:48:46.659609+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T10:07:16.240297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":776,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:47.760789+0000","last_change":"2024-04-19T01:42:22.132251+0000","last_active":"2024-04-19T01:48:47.760789+0000","last_peered":"2024-04-19T01:48:47.760789+0000","last_clean":"2024-04-19T01:48:47.760789+0000","last_became_active":"2024-04-19T01:42:22.131432+0000","last_became_peered":"2024-04-19T01:42:22.131432+0000","last_unstale":"2024-04-19T01:48:47.760789+0000","last_undegraded":"2024-04-19T01:48:47.760789+0000","last_fullsized":"2024-04-19T01:48:47.760789+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:21.102542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:21.102542+0000","last_clean_scrub_stamp":"2024-04-19T01:42:21.102542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T06:47:12.046147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":101,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":101,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607633,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34112,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337728,"statfs":{"total":96636764160,"available":96601833472,"internally_reserved":0,"allocated":3002368,"data_stored":1087303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640336,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34112,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337728,"statfs":{"total":96636764160,"available":96601833472,"internally_reserved":0,"allocated":3002368,"data_stored":1087303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640337,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30476,"kb_used_data":3452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341364,"statfs":{"total":96636764160,"available":96605556736,"internally_reserved":0,"allocated":3534848,"data_stored":1616231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640337,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29956,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341884,"statfs":{"total":96636764160,"available":96606089216,"internally_reserved":0,"allocated":3002368,"data_stored":1087303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640337,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29952,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341888,"statfs":{"total":96636764160,"available":96606093312,"internally_reserved":0,"allocated":3002368,"data_stored":1087303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640338,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34636,"kb_used_data":3452,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337204,"statfs":{"total":96636764160,"available":96601296896,"internally_reserved":0,"allocated":3534848,"data_stored":1616231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:48:55.092 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-19T01:48:55.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.213+0000 7f4e68360640 10 monclient: get_monmap_and_config 2024-04-19T01:48:55.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.213+0000 7f4e68360640 10 monclient: build_initial_monmap 2024-04-19T01:48:55.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.213+0000 7f4e68360640 10 monclient: monmap: 2024-04-19T01:48:55.211 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:55.211 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:55.211 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:48:55.217499+0000 2024-04-19T01:48:55.211 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:48:55.217499+0000 2024-04-19T01:48:55.212 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:55.212 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:55.212 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:48:55.212 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:48:55.212 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:48:55.212 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:48:55.212 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:48:55.212 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:55.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e68360640 1 Processor -- start 2024-04-19T01:48:55.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e68360640 1 -- start start 2024-04-19T01:48:55.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e68360640 10 monclient: init 2024-04-19T01:48:55.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e68360640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:55.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e68360640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-04-19T01:48:55.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e68360640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4e6009f520 addr v1:172.21.15.62:6789/0 2024-04-19T01:48:55.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e68360640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4e60072ed0 0x7f4e60073330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:55.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e68360640 10 monclient(hunting): picked mon.noname-a con 0x7f4e60072ed0 addr v2:172.21.15.62:3300/0 2024-04-19T01:48:55.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e68360640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4e6013a520 0x7f4e6013c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:55.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e68360640 10 monclient(hunting): picked mon.noname-b con 0x7f4e6013a520 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:55.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e68360640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e60132400 con 0x7f4e6009f520 2024-04-19T01:48:55.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e68360640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:55.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e68360640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f4e600a0fc0 con 0x7f4e60072ed0 2024-04-19T01:48:55.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e68360640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:55.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e68360640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f4e6009f900 con 0x7f4e6013a520 2024-04-19T01:48:55.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e68360640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:55.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e68360640 10 monclient(hunting): authenticate will time out at 1321.037129s 2024-04-19T01:48:55.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e658fb640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4e60072ed0 0x7f4e60073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:55.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e658fb640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f4e60072ed0 0x7f4e60073330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47046/0 (socket says 172.21.15.62:47046) 2024-04-19T01:48:55.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e658fb640 1 -- 172.21.15.62:0/277250084 learned_addr learned my addr 172.21.15.62:0/277250084 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:55.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e668fd640 1 --2- 172.21.15.62:0/277250084 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4e6013a520 0x7f4e6013c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:55.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e660fc640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f4e6009f520 0x7f4e60131cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53240/0 (socket says 172.21.15.62:53240) 2024-04-19T01:48:55.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e658fb640 10 monclient(hunting): get_auth_request con 0x7f4e60072ed0 auth_method 0 2024-04-19T01:48:55.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e658fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:55.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e658fb640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:55.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e658fb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:55.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e668fd640 10 monclient(hunting): get_auth_request con 0x7f4e6013a520 auth_method 0 2024-04-19T01:48:55.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e668fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:55.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e668fd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:55.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e668fd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:55.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e658fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:55.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e658fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:55.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e658fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:55.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e668fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:55.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e668fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:55.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e668fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:55.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e658fb640 10 monclient(hunting): handle_auth_done global_id 15303 payload 274 2024-04-19T01:48:55.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e658fb640 1 -- 172.21.15.62:0/277250084 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4e6013a520 msgr2=0x7f4e6013c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:55.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e658fb640 1 --2- 172.21.15.62:0/277250084 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4e6013a520 0x7f4e6013c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:55.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e658fb640 1 -- 172.21.15.62:0/277250084 >> v1:172.21.15.62:6789/0 conn(0x7f4e6009f520 legacy=0x7f4e60131cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:55.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e658fb640 10 monclient: _finish_hunting 0 2024-04-19T01:48:55.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e658fb640 1 monclient: found mon.noname-a 2024-04-19T01:48:55.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e658fb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:55.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e658fb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:48:55.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.217+0000 7f4e658fb640 1 -- 172.21.15.62:0/277250084 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e600a1e70 con 0x7f4e60072ed0 2024-04-19T01:48:55.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e658fb640 1 -- 172.21.15.62:0/277250084 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e60072ed0 msgr2=0x7f4e60073330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:55.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e658fb640 1 --2- 172.21.15.62:0/277250084 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e60072ed0 0x7f4e60073330 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f4e50009870 tx=0x7f4e5001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4ae565e909aad0d server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:55.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e650fa640 1 -- 172.21.15.62:0/277250084 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e500ab620 con 0x7f4e60072ed0 2024-04-19T01:48:55.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e650fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:55.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e650fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:48:55.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e650fa640 10 monclient: dump: 2024-04-19T01:48:55.225 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:55.225 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:55.225 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:55.225 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:55.225 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:55.225 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:55.225 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:55.226 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:55.226 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:55.226 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:55.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e650fa640 10 monclient: _finish_auth 0 2024-04-19T01:48:55.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e650fa640 10 monclient: _check_auth_tickets 2024-04-19T01:48:55.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e650fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:55.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e650fa640 1 -- 172.21.15.62:0/277250084 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e500b0070 con 0x7f4e60072ed0 2024-04-19T01:48:55.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e650fa640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:55.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e650fa640 1 -- 172.21.15.62:0/277250084 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e500ab9e0 con 0x7f4e60072ed0 2024-04-19T01:48:55.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e68360640 5 monclient: authenticate success, global_id 15303 2024-04-19T01:48:55.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e68360640 10 monclient: get_monmap_and_config success 2024-04-19T01:48:55.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e68360640 4 set_mon_vals no callback set 2024-04-19T01:48:55.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e650fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:55.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e650fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:48:55.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e650fa640 10 monclient: dump: 2024-04-19T01:48:55.227 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:55.227 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:55.227 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:55.227 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:55.227 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:55.227 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:55.227 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:55.227 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:55.227 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:55.227 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:55.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e68360640 10 monclient: shutdown 2024-04-19T01:48:55.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e68360640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:55.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e68360640 1 -- 172.21.15.62:0/277250084 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e60072ed0 msgr2=0x7f4e60073330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:55.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e68360640 1 --2- 172.21.15.62:0/277250084 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e60072ed0 0x7f4e60073330 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f4e50009870 tx=0x7f4e5001c480 comp rx=0 tx=0).stop 2024-04-19T01:48:55.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e68360640 1 -- 172.21.15.62:0/277250084 shutdown_connections 2024-04-19T01:48:55.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e68360640 1 --2- 172.21.15.62:0/277250084 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4e6013a520 0x7f4e6013c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:55.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e68360640 1 --2- 172.21.15.62:0/277250084 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e60072ed0 0x7f4e60073330 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:55.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e68360640 1 -- 172.21.15.62:0/277250084 >> 172.21.15.62:0/277250084 conn(0x7f4e600a6210 msgr2=0x7f4e600a6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:55.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e68360640 1 -- 172.21.15.62:0/277250084 shutdown_connections 2024-04-19T01:48:55.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e68360640 1 -- 172.21.15.62:0/277250084 wait complete. 2024-04-19T01:48:55.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e68360640 10 monclient: build_initial_monmap 2024-04-19T01:48:55.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e68360640 10 monclient: monmap: 2024-04-19T01:48:55.228 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:55.228 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:48:55.228 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:48:55.228 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:48:55.228 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:55.229 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:55.229 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:48:55.229 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:48:55.229 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:48:55.229 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:55.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e68360640 1 Processor -- start 2024-04-19T01:48:55.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e68360640 1 -- start start 2024-04-19T01:48:55.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.221+0000 7f4e68360640 10 monclient: init 2024-04-19T01:48:55.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e68360640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:55.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e68360640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-19T01:48:55.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e68360640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4e60072ed0 0x7f4e6012b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:55.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e68360640 10 monclient(hunting): picked mon.noname-c con 0x7f4e60072ed0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:55.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e68360640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e6009f520 0x7f4e6012bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:55.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e68360640 10 monclient(hunting): picked mon.noname-b con 0x7f4e6009f520 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:48:55.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e68360640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e6013a520 0x7f4e601cf260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:55.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e68360640 10 monclient(hunting): picked mon.noname-a con 0x7f4e6013a520 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:48:55.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e68360640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:55.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e68360640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6013fa90 con 0x7f4e6013a520 2024-04-19T01:48:55.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e68360640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:55.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e68360640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6013f910 con 0x7f4e6009f520 2024-04-19T01:48:55.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e68360640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:55.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e68360640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f4e6013fc10 con 0x7f4e60072ed0 2024-04-19T01:48:55.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e68360640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:55.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e68360640 10 monclient(hunting): authenticate will time out at 1321.044238s 2024-04-19T01:48:55.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e660fc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4e60072ed0 0x7f4e6012b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:55.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e668fd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e6013a520 0x7f4e601cf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:55.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e660fc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4e60072ed0 0x7f4e6012b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:39920/0 (socket says 172.21.15.62:39920) 2024-04-19T01:48:55.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e660fc640 1 -- 172.21.15.62:0/3303279508 learned_addr learned my addr 172.21.15.62:0/3303279508 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:55.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e660fc640 10 monclient(hunting): get_auth_request con 0x7f4e60072ed0 auth_method 0 2024-04-19T01:48:55.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e660fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:55.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e660fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:55.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e660fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:55.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e668fd640 10 monclient(hunting): get_auth_request con 0x7f4e6013a520 auth_method 0 2024-04-19T01:48:55.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e668fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:55.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e668fd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:55.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e668fd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:55.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e660fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:55.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e660fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:55.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e660fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:55.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e668fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:55.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e668fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:55.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e658fb640 1 --2- 172.21.15.62:0/3303279508 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e6009f520 0x7f4e6012bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:55.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e668fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:55.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e658fb640 10 monclient(hunting): get_auth_request con 0x7f4e6009f520 auth_method 0 2024-04-19T01:48:55.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e658fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:55.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e658fb640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:55.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e658fb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:55.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e660fc640 10 monclient(hunting): handle_auth_done global_id 34139 payload 824 2024-04-19T01:48:55.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e660fc640 1 -- 172.21.15.62:0/3303279508 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e6009f520 msgr2=0x7f4e6012bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:55.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e660fc640 1 --2- 172.21.15.62:0/3303279508 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e6009f520 0x7f4e6012bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:55.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e660fc640 1 -- 172.21.15.62:0/3303279508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e6013a520 msgr2=0x7f4e601cf260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:55.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e660fc640 1 --2- 172.21.15.62:0/3303279508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e6013a520 0x7f4e601cf260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:55.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e660fc640 10 monclient: _finish_hunting 0 2024-04-19T01:48:55.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e660fc640 1 monclient: found mon.noname-c 2024-04-19T01:48:55.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e660fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:55.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e660fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:48:55.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e660fc640 1 -- 172.21.15.62:0/3303279508 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e60138f90 con 0x7f4e60072ed0 2024-04-19T01:48:55.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e658fb640 1 --2- 172.21.15.62:0/3303279508 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e6009f520 0x7f4e6012bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T01:48:55.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e660fc640 1 -- 172.21.15.62:0/3303279508 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4e60072ed0 msgr2=0x7f4e6012b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:55.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e660fc640 1 --2- 172.21.15.62:0/3303279508 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4e60072ed0 0x7f4e6012b8a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c00c460 tx=0x7f4e5c00c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:55.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e56ffd640 1 -- 172.21.15.62:0/3303279508 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c017070 con 0x7f4e60072ed0 2024-04-19T01:48:55.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:55.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e56ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:48:55.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e56ffd640 10 monclient: dump: 2024-04-19T01:48:55.233 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:55.233 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:55.233 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:55.233 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:55.233 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:55.233 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:55.233 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:55.234 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:55.234 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:55.234 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:55.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e56ffd640 10 monclient: _finish_auth 0 2024-04-19T01:48:55.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e56ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:48:55.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:55.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e68360640 5 monclient: authenticate success, global_id 34139 2024-04-19T01:48:55.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e68360640 10 monclient: _renew_subs 2024-04-19T01:48:55.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e56ffd640 1 -- 172.21.15.62:0/3303279508 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e5c005820 con 0x7f4e60072ed0 2024-04-19T01:48:55.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e68360640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:48:55.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e68360640 1 -- 172.21.15.62:0/3303279508 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e600a1390 con 0x7f4e60072ed0 2024-04-19T01:48:55.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e56ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:55.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e56ffd640 1 -- 172.21.15.62:0/3303279508 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c005c10 con 0x7f4e60072ed0 2024-04-19T01:48:55.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:55.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e56ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:48:55.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e56ffd640 10 monclient: dump: 2024-04-19T01:48:55.235 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:55.235 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:55.235 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:55.235 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:55.235 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:55.235 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:55.235 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:55.235 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:55.235 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:55.235 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:55.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e648f9640 4 set_mon_vals no callback set 2024-04-19T01:48:55.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e68360640 10 monclient: _renew_subs 2024-04-19T01:48:55.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e68360640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:48:55.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.225+0000 7f4e68360640 1 -- 172.21.15.62:0/3303279508 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e60137380 con 0x7f4e60072ed0 2024-04-19T01:48:55.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.229+0000 7f4e68360640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:55.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.229+0000 7f4e56ffd640 1 -- 172.21.15.62:0/3303279508 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f4e5c01a070 con 0x7f4e60072ed0 2024-04-19T01:48:55.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.229+0000 7f4e68360640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:55.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.229+0000 7f4e68360640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:48:55.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.229+0000 7f4e68360640 1 -- 172.21.15.62:0/3303279508 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e600692b0 con 0x7f4e60072ed0 2024-04-19T01:48:55.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.229+0000 7f4e56ffd640 1 --2- 172.21.15.62:0/3303279508 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4e44036420 0x7f4e440388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:55.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.229+0000 7f4e56ffd640 1 -- 172.21.15.62:0/3303279508 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(325..325 src has 115..325) ==== 4346+0+0 (secure 0 0 0) 0x7f4e5c048750 con 0x7f4e60072ed0 2024-04-19T01:48:55.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.229+0000 7f4e658fb640 1 --2- 172.21.15.62:0/3303279508 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4e44036420 0x7f4e440388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:55.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.229+0000 7f4e658fb640 10 monclient: get_auth_request con 0x7f4e44036420 auth_method 0 2024-04-19T01:48:55.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.229+0000 7f4e658fb640 1 --2- 172.21.15.62:0/3303279508 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4e44036420 0x7f4e440388e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f4e6009d090 tx=0x7f4e500a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:55.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.233+0000 7f4e56ffd640 1 -- 172.21.15.62:0/3303279508 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4e5c010300 con 0x7f4e60072ed0 2024-04-19T01:48:55.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.233+0000 7f4e56ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:55.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.233+0000 7f4e56ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:48:55.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.401+0000 7f4e68360640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-19T01:48:55.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.401+0000 7f4e68360640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-19T01:48:55.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.401+0000 7f4e68360640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:48:55.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.401+0000 7f4e68360640 1 -- 172.21.15.62:0/3303279508 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4e6012c320 con 0x7f4e60072ed0 2024-04-19T01:48:55.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.401+0000 7f4e56ffd640 1 -- 172.21.15.62:0/3303279508 <== mon.2 v2:172.21.15.62:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v325) ==== 74+0+12208 (secure 0 0 0) 0x7f4e5c047300 con 0x7f4e60072ed0 2024-04-19T01:48:55.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.401+0000 7f4e56ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-19T01:48:55.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.401+0000 7f4e56ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:48:55.399 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:48:55.399 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":325,"fsid":"6466e4b6-2520-485f-95b3-9401915fbbad","created":"2024-04-19T01:42:09.115311+0000","modified":"2024-04-19T01:48:46.523127+0000","last_up_change":"2024-04-19T01:42:21.102542+0000","last_in_change":"2024-04-19T01:42:12.761455+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":100,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-19T01:42:20.748429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-19T01:42:22.720005+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"64304e60-9fe6-413b-8559-3b8ebd179a7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":322,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":3870805292},{"type":"v1","addr":"172.21.15.62:6802","nonce":3870805292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":3870805292},{"type":"v1","addr":"172.21.15.62:6806","nonce":3870805292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":3870805292},{"type":"v1","addr":"172.21.15.62:6814","nonce":3870805292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":3870805292},{"type":"v1","addr":"172.21.15.62:6810","nonce":3870805292}]},"public_addr":"172.21.15.62:6802/3870805292","cluster_addr":"172.21.15.62:6806/3870805292","heartbeat_back_addr":"172.21.15.62:6814/3870805292","heartbeat_front_addr":"172.21.15.62:6810/3870805292","state":["exists","up"]},{"osd":1,"uuid":"dec72cc0-fa6d-4fe9-b9cd-19dec591ab68","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":322,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":199109455},{"type":"v1","addr":"172.21.15.62:6817","nonce":199109455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":199109455},{"type":"v1","addr":"172.21.15.62:6819","nonce":199109455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":199109455},{"type":"v1","addr":"172.21.15.62:6823","nonce":199109455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":199109455},{"type":"v1","addr":"172.21.15.62:6821","nonce":199109455}]},"public_addr":"172.21.15.62:6817/199109455","cluster_addr":"172.21.15.62:6819/199109455","heartbeat_back_addr":"172.21.15.62:6823/199109455","heartbeat_front_addr":"172.21.15.62:6821/199109455","state":["exists","up"]},{"osd":2,"uuid":"2045906a-e0fd-4c41-b73c-27bf7103243b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":322,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6801","nonce":1128256055},{"type":"v1","addr":"172.21.15.62:6803","nonce":1128256055}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6805","nonce":1128256055},{"type":"v1","addr":"172.21.15.62:6807","nonce":1128256055}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6813","nonce":1128256055},{"type":"v1","addr":"172.21.15.62:6815","nonce":1128256055}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6809","nonce":1128256055},{"type":"v1","addr":"172.21.15.62:6811","nonce":1128256055}]},"public_addr":"172.21.15.62:6803/1128256055","cluster_addr":"172.21.15.62:6807/1128256055","heartbeat_back_addr":"172.21.15.62:6815/1128256055","heartbeat_front_addr":"172.21.15.62:6811/1128256055","state":["exists","up"]},{"osd":3,"uuid":"230f6e15-548c-4338-a8ec-067b8f85bd2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":322,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":2632024688},{"type":"v1","addr":"172.21.15.71:6817","nonce":2632024688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":2632024688},{"type":"v1","addr":"172.21.15.71:6819","nonce":2632024688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":2632024688},{"type":"v1","addr":"172.21.15.71:6823","nonce":2632024688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":2632024688},{"type":"v1","addr":"172.21.15.71:6821","nonce":2632024688}]},"public_addr":"172.21.15.71:6817/2632024688","cluster_addr":"172.21.15.71:6819/2632024688","heartbeat_back_addr":"172.21.15.71:6823/2632024688","heartbeat_front_addr":"172.21.15.71:6821/2632024688","state":["exists","up"]},{"osd":4,"uuid":"06789be2-244f-4b6e-ada2-30b6771676af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":322,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":3342848064},{"type":"v1","addr":"172.21.15.71:6809","nonce":3342848064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3342848064},{"type":"v1","addr":"172.21.15.71:6811","nonce":3342848064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":3342848064},{"type":"v1","addr":"172.21.15.71:6815","nonce":3342848064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":3342848064},{"type":"v1","addr":"172.21.15.71:6813","nonce":3342848064}]},"public_addr":"172.21.15.71:6809/3342848064","cluster_addr":"172.21.15.71:6811/3342848064","heartbeat_back_addr":"172.21.15.71:6815/3342848064","heartbeat_front_addr":"172.21.15.71:6813/3342848064","state":["exists","up"]},{"osd":5,"uuid":"552be339-732f-4801-b14d-3d9b08e7f1f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":322,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":196330660},{"type":"v1","addr":"172.21.15.71:6801","nonce":196330660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":196330660},{"type":"v1","addr":"172.21.15.71:6803","nonce":196330660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":196330660},{"type":"v1","addr":"172.21.15.71:6807","nonce":196330660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":196330660},{"type":"v1","addr":"172.21.15.71:6805","nonce":196330660}]},"public_addr":"172.21.15.71:6801/196330660","cluster_addr":"172.21.15.71:6803/196330660","heartbeat_back_addr":"172.21.15.71:6807/196330660","heartbeat_front_addr":"172.21.15.71:6805/196330660","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:42:18.255067+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:42:18.295998+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:42:18.304269+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:42:18.274604+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:42:18.338084+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T01:42:18.149308+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T01:48:55.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.405+0000 7f4e68360640 1 -- 172.21.15.62:0/3303279508 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4e44036420 msgr2=0x7f4e440388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:55.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.405+0000 7f4e68360640 1 --2- 172.21.15.62:0/3303279508 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4e44036420 0x7f4e440388e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f4e6009d090 tx=0x7f4e500a7000 comp rx=0 tx=0).stop 2024-04-19T01:48:55.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.405+0000 7f4e68360640 10 monclient: shutdown 2024-04-19T01:48:55.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.405+0000 7f4e68360640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:55.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.405+0000 7f4e68360640 1 -- 172.21.15.62:0/3303279508 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4e60072ed0 msgr2=0x7f4e6012b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:55.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.405+0000 7f4e68360640 1 --2- 172.21.15.62:0/3303279508 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4e60072ed0 0x7f4e6012b8a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c00c460 tx=0x7f4e5c00c920 comp rx=0 tx=0).stop 2024-04-19T01:48:55.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.405+0000 7f4e68360640 1 -- 172.21.15.62:0/3303279508 shutdown_connections 2024-04-19T01:48:55.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.405+0000 7f4e68360640 1 --2- 172.21.15.62:0/3303279508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e6013a520 0x7f4e601cf260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:55.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.405+0000 7f4e68360640 1 --2- 172.21.15.62:0/3303279508 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e6009f520 0x7f4e6012bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:55.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.405+0000 7f4e68360640 1 --2- 172.21.15.62:0/3303279508 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4e44036420 0x7f4e440388e0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:55.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.405+0000 7f4e68360640 1 --2- 172.21.15.62:0/3303279508 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4e60072ed0 0x7f4e6012b8a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:55.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.405+0000 7f4e68360640 1 -- 172.21.15.62:0/3303279508 >> 172.21.15.62:0/3303279508 conn(0x7f4e600a6210 msgr2=0x7f4e60130710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:55.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.405+0000 7f4e68360640 1 -- 172.21.15.62:0/3303279508 shutdown_connections 2024-04-19T01:48:55.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:55.405+0000 7f4e68360640 1 -- 172.21.15.62:0/3303279508 wait complete. 2024-04-19T01:48:56.419 INFO:tasks.ceph:Scrubbing osd.0 2024-04-19T01:48:56.419 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-19T01:48:56.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00d18c1640 10 monclient: get_monmap_and_config 2024-04-19T01:48:56.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00d18c1640 10 monclient: build_initial_monmap 2024-04-19T01:48:56.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00d18c1640 10 monclient: monmap: 2024-04-19T01:48:56.557 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:56.557 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:56.557 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:48:56.563127+0000 2024-04-19T01:48:56.557 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:48:56.563127+0000 2024-04-19T01:48:56.557 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:56.557 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:56.557 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:48:56.557 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:48:56.558 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:48:56.558 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:48:56.558 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:48:56.558 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:56.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00d18c1640 1 Processor -- start 2024-04-19T01:48:56.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00d18c1640 1 -- start start 2024-04-19T01:48:56.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00d18c1640 10 monclient: init 2024-04-19T01:48:56.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00d18c1640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:56.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00d18c1640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-04-19T01:48:56.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00d18c1640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f00cc0a10a0 0x7f00cc0a1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:56.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00d18c1640 10 monclient(hunting): picked mon.noname-a con 0x7f00cc0a10a0 addr v2:172.21.15.62:3300/0 2024-04-19T01:48:56.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00d18c1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f00cc0a19c0 addr v1:172.21.15.71:6789/0 2024-04-19T01:48:56.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00d18c1640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f00cc13af40 0x7f00cc13d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:56.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00d18c1640 10 monclient(hunting): picked mon.noname-c con 0x7f00cc13af40 addr v2:172.21.15.71:3300/0 2024-04-19T01:48:56.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00d18c1640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00cc09e580 con 0x7f00cc0a19c0 2024-04-19T01:48:56.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00d18c1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:56.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00d18c1640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f00cc099150 con 0x7f00cc0a10a0 2024-04-19T01:48:56.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00d18c1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:56.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00d18c1640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f00cc09dfc0 con 0x7f00cc13af40 2024-04-19T01:48:56.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00d18c1640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:56.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00d18c1640 10 monclient(hunting): authenticate will time out at 1322.382835s 2024-04-19T01:48:56.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00caffd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f00cc0a10a0 0x7f00cc0a1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:56.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00caffd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f00cc0a10a0 0x7f00cc0a1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47060/0 (socket says 172.21.15.62:47060) 2024-04-19T01:48:56.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00caffd640 1 -- 172.21.15.62:0/169767045 learned_addr learned my addr 172.21.15.62:0/169767045 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:56.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00caffd640 10 monclient(hunting): get_auth_request con 0x7f00cc0a10a0 auth_method 0 2024-04-19T01:48:56.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:56.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00caffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:56.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00caffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:56.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00caffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:56.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00caffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:56.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00caffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:56.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00cb7fe640 1 --2- 172.21.15.62:0/169767045 >> v2:172.21.15.71:3300/0 conn(0x7f00cc13af40 0x7f00cc13d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:56.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00cb7fe640 10 monclient(hunting): get_auth_request con 0x7f00cc13af40 auth_method 0 2024-04-19T01:48:56.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:56.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00cb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:56.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:56.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00caffd640 10 monclient(hunting): handle_auth_done global_id 15309 payload 274 2024-04-19T01:48:56.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00caffd640 1 -- 172.21.15.62:0/169767045 >> v2:172.21.15.71:3300/0 conn(0x7f00cc13af40 msgr2=0x7f00cc13d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:56.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00caffd640 1 --2- 172.21.15.62:0/169767045 >> v2:172.21.15.71:3300/0 conn(0x7f00cc13af40 0x7f00cc13d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:56.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00caffd640 1 -- 172.21.15.62:0/169767045 >> v1:172.21.15.71:6789/0 conn(0x7f00cc0a19c0 legacy=0x7f00cc139820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:56.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00caffd640 10 monclient: _finish_hunting 0 2024-04-19T01:48:56.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00caffd640 1 monclient: found mon.noname-a 2024-04-19T01:48:56.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:56.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00caffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:48:56.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.561+0000 7f00caffd640 1 -- 172.21.15.62:0/169767045 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00cc09f590 con 0x7f00cc0a10a0 2024-04-19T01:48:56.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.565+0000 7f00caffd640 1 -- 172.21.15.62:0/169767045 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00cc0a10a0 msgr2=0x7f00cc0a1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:56.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.565+0000 7f00caffd640 1 --2- 172.21.15.62:0/169767045 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00cc0a10a0 0x7f00cc0a1480 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f00cc0a3540 tx=0x7f00c401c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4a83a51ae7c4cbd server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:56.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00c9ffb640 1 -- 172.21.15.62:0/169767045 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c40a55c0 con 0x7f00cc0a10a0 2024-04-19T01:48:56.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:56.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00c9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:48:56.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00c9ffb640 10 monclient: dump: 2024-04-19T01:48:56.565 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:56.566 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:56.566 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:56.566 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:56.566 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:56.566 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:56.566 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:56.566 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:56.566 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:56.566 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:56.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00c9ffb640 10 monclient: _finish_auth 0 2024-04-19T01:48:56.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00c9ffb640 10 monclient: _check_auth_tickets 2024-04-19T01:48:56.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:56.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00c9ffb640 1 -- 172.21.15.62:0/169767045 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00c40a5760 con 0x7f00cc0a10a0 2024-04-19T01:48:56.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00c9ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:56.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00c9ffb640 1 -- 172.21.15.62:0/169767045 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c40a5b30 con 0x7f00cc0a10a0 2024-04-19T01:48:56.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 5 monclient: authenticate success, global_id 15309 2024-04-19T01:48:56.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 10 monclient: get_monmap_and_config success 2024-04-19T01:48:56.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 4 set_mon_vals no callback set 2024-04-19T01:48:56.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:56.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00c9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:48:56.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00c9ffb640 10 monclient: dump: 2024-04-19T01:48:56.567 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:56.567 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:56.567 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:56.567 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:56.567 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:56.567 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:56.567 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:56.567 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:56.568 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:56.568 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:56.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 10 monclient: shutdown 2024-04-19T01:48:56.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:56.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 1 -- 172.21.15.62:0/169767045 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00cc0a10a0 msgr2=0x7f00cc0a1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:56.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 1 --2- 172.21.15.62:0/169767045 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00cc0a10a0 0x7f00cc0a1480 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f00cc0a3540 tx=0x7f00c401c8b0 comp rx=0 tx=0).stop 2024-04-19T01:48:56.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 1 -- 172.21.15.62:0/169767045 shutdown_connections 2024-04-19T01:48:56.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 1 --2- 172.21.15.62:0/169767045 >> v2:172.21.15.71:3300/0 conn(0x7f00cc13af40 0x7f00cc13d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:56.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 1 --2- 172.21.15.62:0/169767045 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00cc0a10a0 0x7f00cc0a1480 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:56.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 1 -- 172.21.15.62:0/169767045 >> 172.21.15.62:0/169767045 conn(0x7f00cc0a2190 msgr2=0x7f00cc0a25a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:56.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 1 -- 172.21.15.62:0/169767045 shutdown_connections 2024-04-19T01:48:56.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 1 -- 172.21.15.62:0/169767045 wait complete. 2024-04-19T01:48:56.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 10 monclient: build_initial_monmap 2024-04-19T01:48:56.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 10 monclient: monmap: 2024-04-19T01:48:56.569 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:56.569 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:48:56.569 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:48:56.569 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:48:56.569 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:56.569 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:56.569 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:48:56.569 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:48:56.569 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:48:56.569 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:56.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 1 Processor -- start 2024-04-19T01:48:56.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 1 -- start start 2024-04-19T01:48:56.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 10 monclient: init 2024-04-19T01:48:56.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:56.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-19T01:48:56.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f00cc0a10a0 0x7f00cc099690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:56.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 10 monclient(hunting): picked mon.noname-b con 0x7f00cc0a10a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:48:56.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00cc0a19c0 0x7f00cc099bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:56.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 10 monclient(hunting): picked mon.noname-a con 0x7f00cc0a19c0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:48:56.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00cc13af40 0x7f00cc09a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:56.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 10 monclient(hunting): picked mon.noname-c con 0x7f00cc13af40 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:56.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:56.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f00cc09f0f0 con 0x7f00cc0a19c0 2024-04-19T01:48:56.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:56.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f00cc09ef70 con 0x7f00cc0a10a0 2024-04-19T01:48:56.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:56.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f00cc09d6c0 con 0x7f00cc13af40 2024-04-19T01:48:56.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:56.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00d18c1640 10 monclient(hunting): authenticate will time out at 1322.390314s 2024-04-19T01:48:56.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00ca7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00cc0a19c0 0x7f00cc099bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:56.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00ca7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00cc0a19c0 0x7f00cc099bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47066/0 (socket says 172.21.15.62:47066) 2024-04-19T01:48:56.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00ca7fc640 1 -- 172.21.15.62:0/4155114157 learned_addr learned my addr 172.21.15.62:0/4155114157 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:56.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00ca7fc640 10 monclient(hunting): get_auth_request con 0x7f00cc0a19c0 auth_method 0 2024-04-19T01:48:56.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00ca7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:56.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00ca7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:56.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:56.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00cb7fe640 1 --2- 172.21.15.62:0/4155114157 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00cc13af40 0x7f00cc09a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:56.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00cb7fe640 10 monclient(hunting): get_auth_request con 0x7f00cc13af40 auth_method 0 2024-04-19T01:48:56.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:56.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00cb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:56.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:56.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00ca7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:56.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00ca7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:56.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00ca7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:56.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00caffd640 1 --2- 172.21.15.62:0/4155114157 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f00cc0a10a0 0x7f00cc099690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:56.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:56.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:56.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:56.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00caffd640 10 monclient(hunting): get_auth_request con 0x7f00cc0a10a0 auth_method 0 2024-04-19T01:48:56.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:56.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00caffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:56.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00caffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:56.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00ca7fc640 10 monclient(hunting): handle_auth_done global_id 15312 payload 824 2024-04-19T01:48:56.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00ca7fc640 1 -- 172.21.15.62:0/4155114157 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00cc13af40 msgr2=0x7f00cc09a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:56.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00ca7fc640 1 --2- 172.21.15.62:0/4155114157 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00cc13af40 0x7f00cc09a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:56.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00ca7fc640 1 -- 172.21.15.62:0/4155114157 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f00cc0a10a0 msgr2=0x7f00cc099690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:56.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00ca7fc640 1 --2- 172.21.15.62:0/4155114157 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f00cc0a10a0 0x7f00cc099690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:56.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00ca7fc640 10 monclient: _finish_hunting 0 2024-04-19T01:48:56.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00ca7fc640 1 monclient: found mon.noname-a 2024-04-19T01:48:56.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:56.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00ca7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:48:56.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.569+0000 7f00ca7fc640 1 -- 172.21.15.62:0/4155114157 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00cc09aec0 con 0x7f00cc0a19c0 2024-04-19T01:48:56.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00ca7fc640 1 -- 172.21.15.62:0/4155114157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00cc0a19c0 msgr2=0x7f00cc099bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:56.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00ca7fc640 1 --2- 172.21.15.62:0/4155114157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00cc0a19c0 0x7f00cc099bd0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f00c000c450 tx=0x7f00c000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:56.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00b3fff640 1 -- 172.21.15.62:0/4155114157 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c00119a0 con 0x7f00cc0a19c0 2024-04-19T01:48:56.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:56.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00b3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:48:56.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00b3fff640 10 monclient: dump: 2024-04-19T01:48:56.581 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:56.581 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:56.581 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:56.581 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:56.581 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:56.581 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:56.581 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:56.581 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:56.581 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:56.581 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:56.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00b3fff640 10 monclient: _finish_auth 0 2024-04-19T01:48:56.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00b3fff640 10 monclient: _check_auth_tickets 2024-04-19T01:48:56.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:56.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00b3fff640 1 -- 172.21.15.62:0/4155114157 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00c0017070 con 0x7f00cc0a19c0 2024-04-19T01:48:56.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00d18c1640 5 monclient: authenticate success, global_id 15312 2024-04-19T01:48:56.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00d18c1640 10 monclient: _renew_subs 2024-04-19T01:48:56.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00d18c1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:56.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00d18c1640 1 -- 172.21.15.62:0/4155114157 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00cc072a40 con 0x7f00cc0a19c0 2024-04-19T01:48:56.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00b3fff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:56.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00b3fff640 1 -- 172.21.15.62:0/4155114157 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c0011da0 con 0x7f00cc0a19c0 2024-04-19T01:48:56.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:56.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00b3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:48:56.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00b3fff640 10 monclient: dump: 2024-04-19T01:48:56.582 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:56.582 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:56.582 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:56.583 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:56.583 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:56.583 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:56.583 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:56.583 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:56.583 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:56.583 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:56.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00c97fa640 4 set_mon_vals no callback set 2024-04-19T01:48:56.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00d18c1640 10 monclient: _renew_subs 2024-04-19T01:48:56.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00d18c1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:56.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00d18c1640 1 -- 172.21.15.62:0/4155114157 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00cc09aa00 con 0x7f00cc0a19c0 2024-04-19T01:48:56.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00d18c1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:56.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00d18c1640 1 -- 172.21.15.62:0/4155114157 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f00cc04fcf0 con 0x7f00cc0a19c0 2024-04-19T01:48:56.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00b3fff640 1 -- 172.21.15.62:0/4155114157 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f00c001c460 con 0x7f00cc0a19c0 2024-04-19T01:48:56.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00b3fff640 1 --2- 172.21.15.62:0/4155114157 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f00b8036420 0x7f00b80388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:56.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00b3fff640 1 -- 172.21.15.62:0/4155114157 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(325..325 src has 115..325) ==== 4346+0+0 (secure 0 0 0) 0x7f00c0049180 con 0x7f00cc0a19c0 2024-04-19T01:48:56.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00b3fff640 1 --2- 172.21.15.62:0/4155114157 >> [v2:172.21.15.62:6800/3870805292,v1:172.21.15.62:6802/3870805292] conn(0x7f00b803c220 0x7f00b803e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:56.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00b3fff640 1 -- 172.21.15.62:0/4155114157 --> [v2:172.21.15.62:6800/3870805292,v1:172.21.15.62:6802/3870805292] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f00c0009e50 con 0x7f00b803c220 2024-04-19T01:48:56.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00b3fff640 1 -- 172.21.15.62:0/4155114157 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=325) ==== 24+0+0 (secure 0 0 0) 0x7f00c0050050 con 0x7f00cc0a19c0 2024-04-19T01:48:56.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00b3fff640 10 monclient: handle_get_version_reply finishing 1 version 325 2024-04-19T01:48:56.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00cb7fe640 1 --2- 172.21.15.62:0/4155114157 >> [v2:172.21.15.62:6800/3870805292,v1:172.21.15.62:6802/3870805292] conn(0x7f00b803c220 0x7f00b803e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:56.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00cb7fe640 10 monclient: get_auth_request con 0x7f00b803c220 auth_method 0 2024-04-19T01:48:56.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00caffd640 1 --2- 172.21.15.62:0/4155114157 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f00b8036420 0x7f00b80388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:56.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.573+0000 7f00caffd640 10 monclient: get_auth_request con 0x7f00b8036420 auth_method 0 2024-04-19T01:48:56.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.577+0000 7f00caffd640 1 --2- 172.21.15.62:0/4155114157 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f00b8036420 0x7f00b80388e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f00cc076a80 tx=0x7f00c40a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:56.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.577+0000 7f00cb7fe640 1 --2- 172.21.15.62:0/4155114157 >> [v2:172.21.15.62:6800/3870805292,v1:172.21.15.62:6802/3870805292] conn(0x7f00b803c220 0x7f00b803e640 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:56.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.581+0000 7f00b3fff640 1 -- 172.21.15.62:0/4155114157 <== osd.0 v2:172.21.15.62:6800/3870805292 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f00c0009e50 con 0x7f00b803c220 2024-04-19T01:48:56.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.593+0000 7f00d18c1640 1 -- 172.21.15.62:0/4155114157 --> [v2:172.21.15.62:6800/3870805292,v1:172.21.15.62:6802/3870805292] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f00cc0721b0 con 0x7f00b803c220 2024-04-19T01:48:56.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.593+0000 7f00b3fff640 1 -- 172.21.15.62:0/4155114157 <== osd.0 v2:172.21.15.62:6800/3870805292 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f00cc0721b0 con 0x7f00b803c220 2024-04-19T01:48:56.589 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-19T01:48:56.589 INFO:teuthology.orchestra.run.smithi062.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T01:48:56.589 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-19T01:48:56.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.593+0000 7f00d18c1640 1 -- 172.21.15.62:0/4155114157 >> [v2:172.21.15.62:6800/3870805292,v1:172.21.15.62:6802/3870805292] conn(0x7f00b803c220 msgr2=0x7f00b803e640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:56.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.593+0000 7f00d18c1640 1 --2- 172.21.15.62:0/4155114157 >> [v2:172.21.15.62:6800/3870805292,v1:172.21.15.62:6802/3870805292] conn(0x7f00b803c220 0x7f00b803e640 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:56.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.593+0000 7f00d18c1640 1 -- 172.21.15.62:0/4155114157 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f00b8036420 msgr2=0x7f00b80388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:56.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.593+0000 7f00d18c1640 1 --2- 172.21.15.62:0/4155114157 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f00b8036420 0x7f00b80388e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f00cc076a80 tx=0x7f00c40a7070 comp rx=0 tx=0).stop 2024-04-19T01:48:56.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.593+0000 7f00d18c1640 10 monclient: shutdown 2024-04-19T01:48:56.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.593+0000 7f00d18c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:56.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.593+0000 7f00d18c1640 1 -- 172.21.15.62:0/4155114157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00cc0a19c0 msgr2=0x7f00cc099bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:56.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.593+0000 7f00d18c1640 1 --2- 172.21.15.62:0/4155114157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00cc0a19c0 0x7f00cc099bd0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f00c000c450 tx=0x7f00c000c910 comp rx=0 tx=0).stop 2024-04-19T01:48:56.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.593+0000 7f00cb7fe640 1 -- 172.21.15.62:0/4155114157 reap_dead start 2024-04-19T01:48:56.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.593+0000 7f00d18c1640 1 -- 172.21.15.62:0/4155114157 shutdown_connections 2024-04-19T01:48:56.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.593+0000 7f00d18c1640 1 -- 172.21.15.62:0/4155114157 >> 172.21.15.62:0/4155114157 conn(0x7f00cc0a2190 msgr2=0x7f00cc0a4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:56.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.593+0000 7f00d18c1640 1 -- 172.21.15.62:0/4155114157 shutdown_connections 2024-04-19T01:48:56.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.593+0000 7f00d18c1640 1 -- 172.21.15.62:0/4155114157 wait complete. 2024-04-19T01:48:56.605 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-19T01:48:56.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.721+0000 7fbe57430640 10 monclient: get_monmap_and_config 2024-04-19T01:48:56.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.721+0000 7fbe57430640 10 monclient: build_initial_monmap 2024-04-19T01:48:56.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.721+0000 7fbe57430640 10 monclient: monmap: 2024-04-19T01:48:56.718 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:56.719 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:56.719 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:48:56.724718+0000 2024-04-19T01:48:56.719 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:48:56.724718+0000 2024-04-19T01:48:56.719 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:56.719 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:56.719 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:48:56.719 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:48:56.719 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:48:56.720 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:48:56.720 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:48:56.720 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:56.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.721+0000 7fbe57430640 1 Processor -- start 2024-04-19T01:48:56.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.721+0000 7fbe57430640 1 -- start start 2024-04-19T01:48:56.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.721+0000 7fbe57430640 10 monclient: init 2024-04-19T01:48:56.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe57430640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:56.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe57430640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-04-19T01:48:56.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe57430640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe500a47e0 0x7fbe500a4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:56.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe57430640 10 monclient(hunting): picked mon.noname-b con 0x7fbe500a47e0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:56.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe57430640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbe500a5100 0x7fbe501350b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:56.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe57430640 10 monclient(hunting): picked mon.noname-a con 0x7fbe500a5100 addr v2:172.21.15.62:3300/0 2024-04-19T01:48:56.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe57430640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbe501355f0 addr v1:172.21.15.71:6789/0 2024-04-19T01:48:56.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe57430640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe5009f7c0 con 0x7fbe501355f0 2024-04-19T01:48:56.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe57430640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:56.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe57430640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fbe5009bf00 con 0x7fbe500a5100 2024-04-19T01:48:56.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe57430640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:56.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe57430640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fbe5009ee90 con 0x7fbe500a47e0 2024-04-19T01:48:56.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe57430640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:56.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe57430640 10 monclient(hunting): authenticate will time out at 1322.544240s 2024-04-19T01:48:56.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe549cb640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbe500a5100 0x7fbe501350b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:56.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe549cb640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbe500a5100 0x7fbe501350b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47076/0 (socket says 172.21.15.62:47076) 2024-04-19T01:48:56.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe549cb640 1 -- 172.21.15.62:0/2993418532 learned_addr learned my addr 172.21.15.62:0/2993418532 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:56.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe549cb640 10 monclient(hunting): get_auth_request con 0x7fbe500a5100 auth_method 0 2024-04-19T01:48:56.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe549cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:56.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe549cb640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:56.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe549cb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:56.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe551cc640 1 --2- 172.21.15.62:0/2993418532 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe500a47e0 0x7fbe500a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:56.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe551cc640 10 monclient(hunting): get_auth_request con 0x7fbe500a47e0 auth_method 0 2024-04-19T01:48:56.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe551cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:56.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe551cc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:56.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe551cc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:56.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe549cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:56.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe549cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:56.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe549cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:56.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe551cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:56.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe551cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:56.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe551cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:56.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe549cb640 10 monclient(hunting): handle_auth_done global_id 15315 payload 274 2024-04-19T01:48:56.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe549cb640 1 -- 172.21.15.62:0/2993418532 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe500a47e0 msgr2=0x7fbe500a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:56.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe549cb640 1 --2- 172.21.15.62:0/2993418532 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe500a47e0 0x7fbe500a4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:56.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe549cb640 1 -- 172.21.15.62:0/2993418532 >> v1:172.21.15.71:6789/0 conn(0x7fbe501355f0 legacy=0x7fbe5013c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:56.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe549cb640 10 monclient: _finish_hunting 0 2024-04-19T01:48:56.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe549cb640 1 monclient: found mon.noname-a 2024-04-19T01:48:56.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe549cb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:56.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe549cb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:48:56.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.725+0000 7fbe549cb640 1 -- 172.21.15.62:0/2993418532 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe5009c2d0 con 0x7fbe500a5100 2024-04-19T01:48:56.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe549cb640 1 -- 172.21.15.62:0/2993418532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe500a5100 msgr2=0x7fbe501350b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:56.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe549cb640 1 --2- 172.21.15.62:0/2993418532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe500a5100 0x7fbe501350b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fbe40009870 tx=0x7fbe4001c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71964ee9bfd64a66 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:56.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe47fff640 1 -- 172.21.15.62:0/2993418532 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe400b0070 con 0x7fbe500a5100 2024-04-19T01:48:56.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe47fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:56.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe47fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:48:56.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe47fff640 10 monclient: dump: 2024-04-19T01:48:56.732 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:56.732 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:56.732 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:56.732 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:56.732 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:56.732 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:56.732 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:56.732 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:56.733 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:56.733 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:56.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe47fff640 10 monclient: _finish_auth 0 2024-04-19T01:48:56.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe47fff640 10 monclient: _check_auth_tickets 2024-04-19T01:48:56.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe47fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:56.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe47fff640 1 -- 172.21.15.62:0/2993418532 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe400ab600 con 0x7fbe500a5100 2024-04-19T01:48:56.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 5 monclient: authenticate success, global_id 15315 2024-04-19T01:48:56.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:48:56.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe47fff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:56.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 10 monclient: get_monmap_and_config success 2024-04-19T01:48:56.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 4 set_mon_vals no callback set 2024-04-19T01:48:56.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe47fff640 1 -- 172.21.15.62:0/2993418532 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe400ab9d0 con 0x7fbe500a5100 2024-04-19T01:48:56.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe47fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:56.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe47fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:48:56.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe47fff640 10 monclient: dump: 2024-04-19T01:48:56.734 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:56.734 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:56.734 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:56.734 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:56.734 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:56.734 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:56.734 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:56.734 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:56.734 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:56.734 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:56.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 10 monclient: shutdown 2024-04-19T01:48:56.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:56.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 1 -- 172.21.15.62:0/2993418532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe500a5100 msgr2=0x7fbe501350b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:56.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 1 --2- 172.21.15.62:0/2993418532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe500a5100 0x7fbe501350b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fbe40009870 tx=0x7fbe4001c2d0 comp rx=0 tx=0).stop 2024-04-19T01:48:56.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 1 -- 172.21.15.62:0/2993418532 shutdown_connections 2024-04-19T01:48:56.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 1 --2- 172.21.15.62:0/2993418532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe500a5100 0x7fbe501350b0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:56.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 1 --2- 172.21.15.62:0/2993418532 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe500a47e0 0x7fbe500a4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:56.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 1 -- 172.21.15.62:0/2993418532 >> 172.21.15.62:0/2993418532 conn(0x7fbe500a23d0 msgr2=0x7fbe500a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:56.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 1 -- 172.21.15.62:0/2993418532 shutdown_connections 2024-04-19T01:48:56.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 1 -- 172.21.15.62:0/2993418532 wait complete. 2024-04-19T01:48:56.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 10 monclient: build_initial_monmap 2024-04-19T01:48:56.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 10 monclient: monmap: 2024-04-19T01:48:56.735 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:56.735 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:48:56.735 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:48:56.736 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:48:56.736 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:56.736 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:56.736 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:48:56.736 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:48:56.736 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:48:56.736 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:56.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 1 Processor -- start 2024-04-19T01:48:56.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 1 -- start start 2024-04-19T01:48:56.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 10 monclient: init 2024-04-19T01:48:56.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:56.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-19T01:48:56.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe500a47e0 0x7fbe50131bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:56.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 10 monclient(hunting): picked mon.noname-a con 0x7fbe500a47e0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:48:56.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe500a5100 0x7fbe5013ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:56.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 10 monclient(hunting): picked mon.noname-b con 0x7fbe500a5100 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:48:56.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe501355f0 0x7fbe5009cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:56.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 10 monclient(hunting): picked mon.noname-c con 0x7fbe501355f0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:56.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:56.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbe5009f1d0 con 0x7fbe500a47e0 2024-04-19T01:48:56.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:56.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fbe5009f050 con 0x7fbe500a5100 2024-04-19T01:48:56.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:56.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fbe5009c760 con 0x7fbe501355f0 2024-04-19T01:48:56.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:56.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe57430640 10 monclient(hunting): authenticate will time out at 1322.551437s 2024-04-19T01:48:56.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe551cc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe500a47e0 0x7fbe50131bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:56.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe551cc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe500a47e0 0x7fbe50131bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47082/0 (socket says 172.21.15.62:47082) 2024-04-19T01:48:56.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe551cc640 1 -- 172.21.15.62:0/2677392915 learned_addr learned my addr 172.21.15.62:0/2677392915 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:56.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe551cc640 10 monclient(hunting): get_auth_request con 0x7fbe500a47e0 auth_method 0 2024-04-19T01:48:56.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe551cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:56.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe551cc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:56.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe551cc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:56.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe559cd640 1 --2- 172.21.15.62:0/2677392915 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe501355f0 0x7fbe5009cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:56.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe559cd640 10 monclient(hunting): get_auth_request con 0x7fbe501355f0 auth_method 0 2024-04-19T01:48:56.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe559cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:56.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe559cd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:56.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe559cd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:56.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe551cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:56.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe551cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:56.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe551cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:56.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe559cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:56.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe559cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:56.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe559cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:56.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe551cc640 10 monclient(hunting): handle_auth_done global_id 15318 payload 824 2024-04-19T01:48:56.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe549cb640 1 --2- 172.21.15.62:0/2677392915 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe500a5100 0x7fbe5013ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:56.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe551cc640 1 -- 172.21.15.62:0/2677392915 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe501355f0 msgr2=0x7fbe5009cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:56.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe551cc640 1 --2- 172.21.15.62:0/2677392915 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe501355f0 0x7fbe5009cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:56.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe551cc640 1 -- 172.21.15.62:0/2677392915 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe500a5100 msgr2=0x7fbe5013ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:56.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe551cc640 1 --2- 172.21.15.62:0/2677392915 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe500a5100 0x7fbe5013ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:56.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe551cc640 10 monclient: _finish_hunting 0 2024-04-19T01:48:56.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe551cc640 1 monclient: found mon.noname-a 2024-04-19T01:48:56.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe551cc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:56.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe551cc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:48:56.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe551cc640 1 -- 172.21.15.62:0/2677392915 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe5013a830 con 0x7fbe500a47e0 2024-04-19T01:48:56.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe549cb640 10 monclient: get_auth_request con 0x7fbe500a5100 auth_method 0 2024-04-19T01:48:56.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.729+0000 7fbe549cb640 1 --2- 172.21.15.62:0/2677392915 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe500a5100 0x7fbe5013ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:48:56.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.733+0000 7fbe551cc640 1 -- 172.21.15.62:0/2677392915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe500a47e0 msgr2=0x7fbe50131bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:56.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.733+0000 7fbe551cc640 1 --2- 172.21.15.62:0/2677392915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe500a47e0 0x7fbe50131bd0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c00efc0 tx=0x7fbe4c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:56.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.733+0000 7fbe45ffb640 1 -- 172.21.15.62:0/2677392915 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c018020 con 0x7fbe500a47e0 2024-04-19T01:48:56.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.733+0000 7fbe45ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:56.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.733+0000 7fbe45ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:48:56.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.733+0000 7fbe45ffb640 10 monclient: dump: 2024-04-19T01:48:56.742 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:56.742 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:56.742 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:56.742 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:56.742 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:56.742 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:56.742 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:56.742 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:56.742 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:56.742 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:56.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.733+0000 7fbe45ffb640 10 monclient: _finish_auth 0 2024-04-19T01:48:56.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.733+0000 7fbe45ffb640 10 monclient: _check_auth_tickets 2024-04-19T01:48:56.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.733+0000 7fbe45ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:56.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.733+0000 7fbe45ffb640 1 -- 172.21.15.62:0/2677392915 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe4c014680 con 0x7fbe500a47e0 2024-04-19T01:48:56.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.733+0000 7fbe45ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:56.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.733+0000 7fbe45ffb640 1 -- 172.21.15.62:0/2677392915 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c014a50 con 0x7fbe500a47e0 2024-04-19T01:48:56.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.733+0000 7fbe57430640 5 monclient: authenticate success, global_id 15318 2024-04-19T01:48:56.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.733+0000 7fbe57430640 10 monclient: _renew_subs 2024-04-19T01:48:56.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.733+0000 7fbe57430640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:56.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.733+0000 7fbe57430640 1 -- 172.21.15.62:0/2677392915 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe5013ab10 con 0x7fbe500a47e0 2024-04-19T01:48:56.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.733+0000 7fbe45ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:56.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.733+0000 7fbe45ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:48:56.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.733+0000 7fbe45ffb640 10 monclient: dump: 2024-04-19T01:48:56.743 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:56.743 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:56.743 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:56.743 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:56.744 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:56.744 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:56.744 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:56.744 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:56.744 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:56.744 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:56.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.733+0000 7fbe477fe640 4 set_mon_vals no callback set 2024-04-19T01:48:56.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.733+0000 7fbe57430640 10 monclient: _renew_subs 2024-04-19T01:48:56.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.733+0000 7fbe57430640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:56.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.733+0000 7fbe57430640 1 -- 172.21.15.62:0/2677392915 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe5009e850 con 0x7fbe500a47e0 2024-04-19T01:48:56.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.733+0000 7fbe57430640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:56.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.733+0000 7fbe57430640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:56.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.733+0000 7fbe57430640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:56.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.733+0000 7fbe57430640 1 -- 172.21.15.62:0/2677392915 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe500692b0 con 0x7fbe500a47e0 2024-04-19T01:48:56.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.733+0000 7fbe45ffb640 1 -- 172.21.15.62:0/2677392915 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fbe4c01b020 con 0x7fbe500a47e0 2024-04-19T01:48:56.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.733+0000 7fbe45ffb640 1 --2- 172.21.15.62:0/2677392915 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fbe34036480 0x7fbe34038940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:56.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.733+0000 7fbe45ffb640 1 -- 172.21.15.62:0/2677392915 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(325..325 src has 115..325) ==== 4346+0+0 (secure 0 0 0) 0x7fbe4c049d80 con 0x7fbe500a47e0 2024-04-19T01:48:56.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.733+0000 7fbe549cb640 1 --2- 172.21.15.62:0/2677392915 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fbe34036480 0x7fbe34038940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:56.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.737+0000 7fbe549cb640 10 monclient: get_auth_request con 0x7fbe34036480 auth_method 0 2024-04-19T01:48:56.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.737+0000 7fbe549cb640 1 --2- 172.21.15.62:0/2677392915 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fbe34036480 0x7fbe34038940 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fbe5009bd70 tx=0x7fbe40008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:56.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.741+0000 7fbe45ffb640 1 -- 172.21.15.62:0/2677392915 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fbe4c022070 con 0x7fbe500a47e0 2024-04-19T01:48:56.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.741+0000 7fbe45ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:56.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.741+0000 7fbe45ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:48:56.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.905+0000 7fbe57430640 1 -- 172.21.15.62:0/2677392915 --> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7fbe50139660 con 0x7fbe34036480 2024-04-19T01:48:56.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.905+0000 7fbe45ffb640 1 -- 172.21.15.62:0/2677392915 <== mgr.4108 v2:172.21.15.71:6824/2133533992 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fbe50139660 con 0x7fbe34036480 2024-04-19T01:48:56.902 INFO:teuthology.orchestra.run.smithi062.stderr:instructed osd(s) 0 to deep-scrub 2024-04-19T01:48:56.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.909+0000 7fbe57430640 1 -- 172.21.15.62:0/2677392915 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fbe34036480 msgr2=0x7fbe34038940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:56.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.909+0000 7fbe57430640 1 --2- 172.21.15.62:0/2677392915 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fbe34036480 0x7fbe34038940 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fbe5009bd70 tx=0x7fbe40008040 comp rx=0 tx=0).stop 2024-04-19T01:48:56.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.909+0000 7fbe57430640 10 monclient: shutdown 2024-04-19T01:48:56.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.909+0000 7fbe57430640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:56.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.909+0000 7fbe57430640 1 -- 172.21.15.62:0/2677392915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe500a47e0 msgr2=0x7fbe50131bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:56.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.909+0000 7fbe57430640 1 --2- 172.21.15.62:0/2677392915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe500a47e0 0x7fbe50131bd0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c00efc0 tx=0x7fbe4c00c490 comp rx=0 tx=0).stop 2024-04-19T01:48:56.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.909+0000 7fbe57430640 1 -- 172.21.15.62:0/2677392915 shutdown_connections 2024-04-19T01:48:56.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.909+0000 7fbe57430640 1 --2- 172.21.15.62:0/2677392915 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fbe34036480 0x7fbe34038940 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:56.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.909+0000 7fbe57430640 1 --2- 172.21.15.62:0/2677392915 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fbe501355f0 0x7fbe5009cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:56.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.909+0000 7fbe57430640 1 --2- 172.21.15.62:0/2677392915 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe500a5100 0x7fbe5013ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:56.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.909+0000 7fbe57430640 1 --2- 172.21.15.62:0/2677392915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe500a47e0 0x7fbe50131bd0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:56.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.909+0000 7fbe57430640 1 -- 172.21.15.62:0/2677392915 >> 172.21.15.62:0/2677392915 conn(0x7fbe500a23d0 msgr2=0x7fbe5006f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:56.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.909+0000 7fbe57430640 1 -- 172.21.15.62:0/2677392915 shutdown_connections 2024-04-19T01:48:56.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:56.909+0000 7fbe57430640 1 -- 172.21.15.62:0/2677392915 wait complete. 2024-04-19T01:48:56.922 INFO:tasks.ceph:Scrubbing osd.1 2024-04-19T01:48:56.922 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-19T01:48:57.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d8b7f3640 10 monclient: get_monmap_and_config 2024-04-19T01:48:57.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d8b7f3640 10 monclient: build_initial_monmap 2024-04-19T01:48:57.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d8b7f3640 10 monclient: monmap: 2024-04-19T01:48:57.052 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:57.052 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:57.053 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:48:57.058721+0000 2024-04-19T01:48:57.053 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:48:57.058721+0000 2024-04-19T01:48:57.053 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:57.053 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:57.053 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:48:57.053 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:48:57.053 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:48:57.053 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:48:57.053 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:48:57.053 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:57.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d8b7f3640 1 Processor -- start 2024-04-19T01:48:57.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d8b7f3640 1 -- start start 2024-04-19T01:48:57.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d8b7f3640 10 monclient: init 2024-04-19T01:48:57.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d8b7f3640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:57.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d8b7f3640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-04-19T01:48:57.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d8b7f3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4d840a10a0 addr v1:172.21.15.71:6789/0 2024-04-19T01:48:57.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d8b7f3640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4d840a1b90 0x7f4d8413a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:57.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d8b7f3640 10 monclient(hunting): picked mon.noname-c con 0x7f4d840a1b90 addr v2:172.21.15.71:3300/0 2024-04-19T01:48:57.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d8b7f3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d8413ad70 0x7f4d8413d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:57.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d8b7f3640 10 monclient(hunting): picked mon.noname-b con 0x7f4d8413ad70 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:57.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d8b7f3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d8409e580 con 0x7f4d840a10a0 2024-04-19T01:48:57.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d8b7f3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:57.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d8b7f3640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f4d84099150 con 0x7f4d840a1b90 2024-04-19T01:48:57.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d8b7f3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:57.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d8b7f3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f4d8409dfc0 con 0x7f4d8413ad70 2024-04-19T01:48:57.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d8b7f3640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:57.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d8b7f3640 10 monclient(hunting): authenticate will time out at 1322.878181s 2024-04-19T01:48:57.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d89d90640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d8413ad70 0x7f4d8413d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:57.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d89d90640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d8413ad70 0x7f4d8413d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:39962/0 (socket says 172.21.15.62:39962) 2024-04-19T01:48:57.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d89d90640 1 -- 172.21.15.62:0/2211945544 learned_addr learned my addr 172.21.15.62:0/2211945544 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:57.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d89d90640 10 monclient(hunting): get_auth_request con 0x7f4d8413ad70 auth_method 0 2024-04-19T01:48:57.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d89d90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:57.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d89d90640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:57.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d89d90640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:57.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d89d90640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:57.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d89d90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:57.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d89d90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:57.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d88d8e640 1 --2- 172.21.15.62:0/2211945544 >> v2:172.21.15.71:3300/0 conn(0x7f4d840a1b90 0x7f4d8413a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:57.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d88d8e640 10 monclient(hunting): get_auth_request con 0x7f4d840a1b90 auth_method 0 2024-04-19T01:48:57.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d88d8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:57.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d88d8e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:57.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d88d8e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:57.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d89d90640 10 monclient(hunting): handle_auth_done global_id 34151 payload 274 2024-04-19T01:48:57.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d89d90640 1 -- 172.21.15.62:0/2211945544 >> v2:172.21.15.71:3300/0 conn(0x7f4d840a1b90 msgr2=0x7f4d8413a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:57.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d89d90640 1 --2- 172.21.15.62:0/2211945544 >> v2:172.21.15.71:3300/0 conn(0x7f4d840a1b90 0x7f4d8413a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d89d90640 1 -- 172.21.15.62:0/2211945544 >> v1:172.21.15.71:6789/0 conn(0x7f4d840a10a0 legacy=0x7f4d840a1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:57.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d89d90640 10 monclient: _finish_hunting 0 2024-04-19T01:48:57.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d89d90640 1 monclient: found mon.noname-b 2024-04-19T01:48:57.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d89d90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:57.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d89d90640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:48:57.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.057+0000 7f4d89d90640 1 -- 172.21.15.62:0/2211945544 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d8409f590 con 0x7f4d8413ad70 2024-04-19T01:48:57.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.061+0000 7f4d89d90640 1 -- 172.21.15.62:0/2211945544 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d8413ad70 msgr2=0x7f4d8413d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:57.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.061+0000 7f4d89d90640 1 --2- 172.21.15.62:0/2211945544 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d8413ad70 0x7f4d8413d160 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c009e50 tx=0x7f4d7c01c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f21196bef950565 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:57.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.061+0000 7f4d7bfff640 1 -- 172.21.15.62:0/2211945544 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c0a5530 con 0x7f4d8413ad70 2024-04-19T01:48:57.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.061+0000 7f4d7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:57.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.061+0000 7f4d7bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:48:57.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.061+0000 7f4d7bfff640 10 monclient: dump: 2024-04-19T01:48:57.061 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:57.061 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:57.061 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:57.061 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:57.061 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:57.061 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:57.061 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:57.061 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:57.061 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:57.061 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:57.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.061+0000 7f4d7bfff640 10 monclient: _finish_auth 0 2024-04-19T01:48:57.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.061+0000 7f4d7bfff640 10 monclient: _check_auth_tickets 2024-04-19T01:48:57.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.061+0000 7f4d7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:57.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.061+0000 7f4d7bfff640 1 -- 172.21.15.62:0/2211945544 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d7c0a56d0 con 0x7f4d8413ad70 2024-04-19T01:48:57.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.061+0000 7f4d7bfff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:57.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.061+0000 7f4d7bfff640 1 -- 172.21.15.62:0/2211945544 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c0a5aa0 con 0x7f4d8413ad70 2024-04-19T01:48:57.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.061+0000 7f4d8b7f3640 5 monclient: authenticate success, global_id 34151 2024-04-19T01:48:57.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.061+0000 7f4d8b7f3640 10 monclient: get_monmap_and_config success 2024-04-19T01:48:57.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.061+0000 7f4d8b7f3640 4 set_mon_vals no callback set 2024-04-19T01:48:57.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.061+0000 7f4d7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:57.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.061+0000 7f4d7bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:48:57.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.061+0000 7f4d7bfff640 10 monclient: dump: 2024-04-19T01:48:57.062 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:57.062 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:57.062 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:57.062 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:57.063 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:57.063 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:57.063 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:57.063 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:57.063 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:57.063 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:57.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8b7f3640 10 monclient: shutdown 2024-04-19T01:48:57.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8b7f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:57.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8b7f3640 1 -- 172.21.15.62:0/2211945544 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d8413ad70 msgr2=0x7f4d8413d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:57.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8b7f3640 1 --2- 172.21.15.62:0/2211945544 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d8413ad70 0x7f4d8413d160 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c009e50 tx=0x7f4d7c01c8f0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8b7f3640 1 -- 172.21.15.62:0/2211945544 shutdown_connections 2024-04-19T01:48:57.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8b7f3640 1 --2- 172.21.15.62:0/2211945544 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d8413ad70 0x7f4d8413d160 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8b7f3640 1 --2- 172.21.15.62:0/2211945544 >> v2:172.21.15.71:3300/0 conn(0x7f4d840a1b90 0x7f4d8413a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8b7f3640 1 -- 172.21.15.62:0/2211945544 >> 172.21.15.62:0/2211945544 conn(0x7f4d840a2190 msgr2=0x7f4d840a25a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:57.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8b7f3640 1 -- 172.21.15.62:0/2211945544 shutdown_connections 2024-04-19T01:48:57.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8b7f3640 1 -- 172.21.15.62:0/2211945544 wait complete. 2024-04-19T01:48:57.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8b7f3640 10 monclient: build_initial_monmap 2024-04-19T01:48:57.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8b7f3640 10 monclient: monmap: 2024-04-19T01:48:57.064 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:57.064 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:48:57.064 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:48:57.064 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:48:57.064 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:57.064 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:57.064 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:48:57.064 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:48:57.064 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:48:57.064 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:57.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8b7f3640 1 Processor -- start 2024-04-19T01:48:57.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8b7f3640 1 -- start start 2024-04-19T01:48:57.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8b7f3640 10 monclient: init 2024-04-19T01:48:57.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8b7f3640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:57.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8b7f3640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-19T01:48:57.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8b7f3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d840a10a0 0x7f4d841cf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:57.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8b7f3640 10 monclient(hunting): picked mon.noname-b con 0x7f4d840a10a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:48:57.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8b7f3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d840a1b90 0x7f4d84098440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:57.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8b7f3640 10 monclient(hunting): picked mon.noname-a con 0x7f4d840a1b90 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:48:57.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8b7f3640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d8413ad70 0x7f4d84098980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:57.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8b7f3640 10 monclient(hunting): picked mon.noname-c con 0x7f4d8413ad70 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:57.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8b7f3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:57.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8b7f3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8409d840 con 0x7f4d840a1b90 2024-04-19T01:48:57.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8b7f3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:57.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8b7f3640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8409d6c0 con 0x7f4d840a10a0 2024-04-19T01:48:57.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8b7f3640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:57.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8b7f3640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f4d8413fec0 con 0x7f4d8413ad70 2024-04-19T01:48:57.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8b7f3640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:57.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8b7f3640 10 monclient(hunting): authenticate will time out at 1322.886073s 2024-04-19T01:48:57.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d88d8e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d840a1b90 0x7f4d84098440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:57.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d89d90640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d8413ad70 0x7f4d84098980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:57.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d88d8e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d840a1b90 0x7f4d84098440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47086/0 (socket says 172.21.15.62:47086) 2024-04-19T01:48:57.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d89d90640 10 monclient(hunting): get_auth_request con 0x7f4d8413ad70 auth_method 0 2024-04-19T01:48:57.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d88d8e640 1 -- 172.21.15.62:0/1972154737 learned_addr learned my addr 172.21.15.62:0/1972154737 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:57.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d89d90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:57.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d89d90640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:57.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d89d90640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:57.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d88d8e640 10 monclient(hunting): get_auth_request con 0x7f4d840a1b90 auth_method 0 2024-04-19T01:48:57.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d88d8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:57.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d88d8e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:57.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d88d8e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:57.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d89d90640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:57.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d89d90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:57.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d89d90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:57.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d88d8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:57.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d88d8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:57.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d88d8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:57.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8958f640 1 --2- 172.21.15.62:0/1972154737 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d840a10a0 0x7f4d841cf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:57.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d89d90640 10 monclient(hunting): handle_auth_done global_id 34154 payload 824 2024-04-19T01:48:57.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d89d90640 1 -- 172.21.15.62:0/1972154737 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d840a10a0 msgr2=0x7f4d841cf5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:57.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d89d90640 1 --2- 172.21.15.62:0/1972154737 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d840a10a0 0x7f4d841cf5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d89d90640 1 -- 172.21.15.62:0/1972154737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d840a1b90 msgr2=0x7f4d84098440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:57.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d89d90640 1 --2- 172.21.15.62:0/1972154737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d840a1b90 0x7f4d84098440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d89d90640 10 monclient: _finish_hunting 0 2024-04-19T01:48:57.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d89d90640 1 monclient: found mon.noname-c 2024-04-19T01:48:57.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d89d90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:57.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d89d90640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:48:57.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d89d90640 1 -- 172.21.15.62:0/1972154737 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d84092ed0 con 0x7f4d8413ad70 2024-04-19T01:48:57.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8958f640 10 monclient: get_auth_request con 0x7f4d840a10a0 auth_method 0 2024-04-19T01:48:57.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.065+0000 7f4d8958f640 1 --2- 172.21.15.62:0/1972154737 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d840a10a0 0x7f4d841cf5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:48:57.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d89d90640 1 -- 172.21.15.62:0/1972154737 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d8413ad70 msgr2=0x7f4d84098980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:57.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d89d90640 1 --2- 172.21.15.62:0/1972154737 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d8413ad70 0x7f4d84098980 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c0ab000 tx=0x7f4d7c019690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:57.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d79ffb640 1 -- 172.21.15.62:0/1972154737 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c0bc070 con 0x7f4d8413ad70 2024-04-19T01:48:57.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d79ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:57.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d79ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:48:57.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d79ffb640 10 monclient: dump: 2024-04-19T01:48:57.073 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:57.073 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:57.073 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:57.073 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:57.073 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:57.073 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:57.073 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:57.073 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:57.073 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:57.073 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:57.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d79ffb640 10 monclient: _finish_auth 0 2024-04-19T01:48:57.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d79ffb640 10 monclient: _check_auth_tickets 2024-04-19T01:48:57.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:57.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d79ffb640 1 -- 172.21.15.62:0/1972154737 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d7c0b8a90 con 0x7f4d8413ad70 2024-04-19T01:48:57.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d8b7f3640 5 monclient: authenticate success, global_id 34154 2024-04-19T01:48:57.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d8b7f3640 10 monclient: _renew_subs 2024-04-19T01:48:57.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d8b7f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:48:57.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d8b7f3640 1 -- 172.21.15.62:0/1972154737 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d84092bf0 con 0x7f4d8413ad70 2024-04-19T01:48:57.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d79ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:57.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d79ffb640 1 -- 172.21.15.62:0/1972154737 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c0a5530 con 0x7f4d8413ad70 2024-04-19T01:48:57.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d79ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:57.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d79ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:48:57.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d79ffb640 10 monclient: dump: 2024-04-19T01:48:57.074 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:57.074 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:57.074 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:57.074 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:57.075 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:57.075 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:57.075 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:57.075 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:57.075 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:57.075 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:57.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d8b7f3640 10 monclient: _renew_subs 2024-04-19T01:48:57.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d7b7fe640 4 set_mon_vals no callback set 2024-04-19T01:48:57.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d8b7f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:48:57.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d8b7f3640 1 -- 172.21.15.62:0/1972154737 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d840915a0 con 0x7f4d8413ad70 2024-04-19T01:48:57.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d8b7f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:48:57.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d8b7f3640 1 -- 172.21.15.62:0/1972154737 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4d841cfb10 con 0x7f4d8413ad70 2024-04-19T01:48:57.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d79ffb640 1 -- 172.21.15.62:0/1972154737 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f4d7c0c0400 con 0x7f4d8413ad70 2024-04-19T01:48:57.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d79ffb640 1 --2- 172.21.15.62:0/1972154737 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4d68036470 0x7f4d68038930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:57.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d79ffb640 1 -- 172.21.15.62:0/1972154737 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(325..325 src has 115..325) ==== 4346+0+0 (secure 0 0 0) 0x7f4d7c0e3aa0 con 0x7f4d8413ad70 2024-04-19T01:48:57.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d79ffb640 1 --2- 172.21.15.62:0/1972154737 >> [v2:172.21.15.62:6816/199109455,v1:172.21.15.62:6817/199109455] conn(0x7f4d6803c2f0 0x7f4d6803e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:57.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d79ffb640 1 -- 172.21.15.62:0/1972154737 --> [v2:172.21.15.62:6816/199109455,v1:172.21.15.62:6817/199109455] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4d7c0b9e50 con 0x7f4d6803c2f0 2024-04-19T01:48:57.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d79ffb640 1 -- 172.21.15.62:0/1972154737 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_get_version_reply(handle=1 version=325) ==== 24+0+0 (secure 0 0 0) 0x7f4d7c0e5e60 con 0x7f4d8413ad70 2024-04-19T01:48:57.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d79ffb640 10 monclient: handle_get_version_reply finishing 1 version 325 2024-04-19T01:48:57.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d88d8e640 1 --2- 172.21.15.62:0/1972154737 >> [v2:172.21.15.62:6816/199109455,v1:172.21.15.62:6817/199109455] conn(0x7f4d6803c2f0 0x7f4d6803e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:57.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d88d8e640 10 monclient: get_auth_request con 0x7f4d6803c2f0 auth_method 0 2024-04-19T01:48:57.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d8958f640 1 --2- 172.21.15.62:0/1972154737 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4d68036470 0x7f4d68038930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:57.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.069+0000 7f4d8958f640 10 monclient: get_auth_request con 0x7f4d68036470 auth_method 0 2024-04-19T01:48:57.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.073+0000 7f4d8958f640 1 --2- 172.21.15.62:0/1972154737 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4d68036470 0x7f4d68038930 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f4d8000a850 tx=0x7f4d80008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:57.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.073+0000 7f4d88d8e640 1 --2- 172.21.15.62:0/1972154737 >> [v2:172.21.15.62:6816/199109455,v1:172.21.15.62:6817/199109455] conn(0x7f4d6803c2f0 0x7f4d6803e710 crc :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:57.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.073+0000 7f4d79ffb640 1 -- 172.21.15.62:0/1972154737 <== osd.1 v2:172.21.15.62:6816/199109455 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4d7c0b9e50 con 0x7f4d6803c2f0 2024-04-19T01:48:57.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.089+0000 7f4d8b7f3640 1 -- 172.21.15.62:0/1972154737 --> [v2:172.21.15.62:6816/199109455,v1:172.21.15.62:6817/199109455] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f4d84091320 con 0x7f4d6803c2f0 2024-04-19T01:48:57.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.089+0000 7f4d79ffb640 1 -- 172.21.15.62:0/1972154737 <== osd.1 v2:172.21.15.62:6816/199109455 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f4d84091320 con 0x7f4d6803c2f0 2024-04-19T01:48:57.085 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-19T01:48:57.085 INFO:teuthology.orchestra.run.smithi062.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T01:48:57.085 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-19T01:48:57.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.089+0000 7f4d8b7f3640 1 -- 172.21.15.62:0/1972154737 >> [v2:172.21.15.62:6816/199109455,v1:172.21.15.62:6817/199109455] conn(0x7f4d6803c2f0 msgr2=0x7f4d6803e710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:57.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.089+0000 7f4d8b7f3640 1 --2- 172.21.15.62:0/1972154737 >> [v2:172.21.15.62:6816/199109455,v1:172.21.15.62:6817/199109455] conn(0x7f4d6803c2f0 0x7f4d6803e710 crc :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.089+0000 7f4d8b7f3640 1 -- 172.21.15.62:0/1972154737 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4d68036470 msgr2=0x7f4d68038930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:57.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.089+0000 7f4d8b7f3640 1 --2- 172.21.15.62:0/1972154737 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f4d68036470 0x7f4d68038930 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f4d8000a850 tx=0x7f4d80008040 comp rx=0 tx=0).stop 2024-04-19T01:48:57.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.089+0000 7f4d8b7f3640 10 monclient: shutdown 2024-04-19T01:48:57.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.089+0000 7f4d8b7f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:57.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.089+0000 7f4d8b7f3640 1 -- 172.21.15.62:0/1972154737 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d8413ad70 msgr2=0x7f4d84098980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:57.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.089+0000 7f4d8b7f3640 1 --2- 172.21.15.62:0/1972154737 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f4d8413ad70 0x7f4d84098980 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c0ab000 tx=0x7f4d7c019690 comp rx=0 tx=0).stop 2024-04-19T01:48:57.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.089+0000 7f4d89d90640 1 -- 172.21.15.62:0/1972154737 reap_dead start 2024-04-19T01:48:57.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.089+0000 7f4d8b7f3640 1 -- 172.21.15.62:0/1972154737 shutdown_connections 2024-04-19T01:48:57.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.089+0000 7f4d8b7f3640 1 -- 172.21.15.62:0/1972154737 >> 172.21.15.62:0/1972154737 conn(0x7f4d840a2190 msgr2=0x7f4d8413b150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:57.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.089+0000 7f4d8b7f3640 1 -- 172.21.15.62:0/1972154737 shutdown_connections 2024-04-19T01:48:57.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.089+0000 7f4d8b7f3640 1 -- 172.21.15.62:0/1972154737 wait complete. 2024-04-19T01:48:57.101 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-19T01:48:57.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa025892640 10 monclient: get_monmap_and_config 2024-04-19T01:48:57.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa025892640 10 monclient: build_initial_monmap 2024-04-19T01:48:57.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa025892640 10 monclient: monmap: 2024-04-19T01:48:57.216 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:57.216 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:57.216 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:48:57.222332+0000 2024-04-19T01:48:57.216 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:48:57.222332+0000 2024-04-19T01:48:57.216 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:57.216 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:57.217 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:48:57.217 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:48:57.217 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:48:57.217 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:48:57.217 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:48:57.217 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:57.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa025892640 1 Processor -- start 2024-04-19T01:48:57.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa025892640 1 -- start start 2024-04-19T01:48:57.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa025892640 10 monclient: init 2024-04-19T01:48:57.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa025892640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:57.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa025892640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-04-19T01:48:57.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa025892640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa0200a3410 0x7fa0200a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:57.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa025892640 10 monclient(hunting): picked mon.noname-c con 0x7fa0200a3410 addr v2:172.21.15.71:3300/0 2024-04-19T01:48:57.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa025892640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa0200a2900 0x7fa0200a46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:57.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa025892640 10 monclient(hunting): picked mon.noname-a con 0x7fa0200a2900 addr v2:172.21.15.62:3300/0 2024-04-19T01:48:57.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa025892640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa0200a2d50 addr v1:172.21.15.71:6789/0 2024-04-19T01:48:57.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa025892640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0200a1430 con 0x7fa0200a2d50 2024-04-19T01:48:57.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa025892640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:57.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa025892640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fa0200a1710 con 0x7fa0200a2900 2024-04-19T01:48:57.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa025892640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:57.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa025892640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fa0200a19f0 con 0x7fa0200a3410 2024-04-19T01:48:57.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa025892640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:57.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa025892640 10 monclient(hunting): authenticate will time out at 1323.041864s 2024-04-19T01:48:57.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa01e7fc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa0200a2900 0x7fa0200a46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:57.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa01e7fc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa0200a2900 0x7fa0200a46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47092/0 (socket says 172.21.15.62:47092) 2024-04-19T01:48:57.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa01e7fc640 1 -- 172.21.15.62:0/376606368 learned_addr learned my addr 172.21.15.62:0/376606368 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:57.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa01e7fc640 10 monclient(hunting): get_auth_request con 0x7fa0200a2900 auth_method 0 2024-04-19T01:48:57.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa01e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:57.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa01e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:57.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa01e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:57.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa01e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:57.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa01e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:57.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa01e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:57.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa01effd640 1 --2- 172.21.15.62:0/376606368 >> v2:172.21.15.71:3300/0 conn(0x7fa0200a3410 0x7fa0200a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:57.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa01effd640 10 monclient(hunting): get_auth_request con 0x7fa0200a3410 auth_method 0 2024-04-19T01:48:57.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa01effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:57.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa01effd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:57.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa01effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:57.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa01e7fc640 10 monclient(hunting): handle_auth_done global_id 15324 payload 274 2024-04-19T01:48:57.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa01e7fc640 1 -- 172.21.15.62:0/376606368 >> v2:172.21.15.71:3300/0 conn(0x7fa0200a3410 msgr2=0x7fa0200a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:57.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa01e7fc640 1 --2- 172.21.15.62:0/376606368 >> v2:172.21.15.71:3300/0 conn(0x7fa0200a3410 0x7fa0200a37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa01e7fc640 1 -- 172.21.15.62:0/376606368 >> v1:172.21.15.71:6789/0 conn(0x7fa0200a2d50 legacy=0x7fa0200a4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:57.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa01e7fc640 10 monclient: _finish_hunting 0 2024-04-19T01:48:57.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa01e7fc640 1 monclient: found mon.noname-a 2024-04-19T01:48:57.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa01e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:57.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa01e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:48:57.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.221+0000 7fa01e7fc640 1 -- 172.21.15.62:0/376606368 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0200a3d30 con 0x7fa0200a2900 2024-04-19T01:48:57.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.225+0000 7fa01e7fc640 1 -- 172.21.15.62:0/376606368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0200a2900 msgr2=0x7fa0200a46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:57.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.225+0000 7fa01e7fc640 1 --2- 172.21.15.62:0/376606368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0200a2900 0x7fa0200a46d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fa020073940 tx=0x7fa01001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5b1f9cfb84abfc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:57.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.225+0000 7fa01dffb640 1 -- 172.21.15.62:0/376606368 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0100b0070 con 0x7fa0200a2900 2024-04-19T01:48:57.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.225+0000 7fa01dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:57.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.225+0000 7fa01dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:48:57.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.225+0000 7fa01dffb640 10 monclient: dump: 2024-04-19T01:48:57.231 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:57.231 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:57.232 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:57.232 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:57.232 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:57.232 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:57.232 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:57.232 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:57.232 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:57.232 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:57.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.225+0000 7fa01dffb640 10 monclient: _finish_auth 0 2024-04-19T01:48:57.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.225+0000 7fa01dffb640 10 monclient: _check_auth_tickets 2024-04-19T01:48:57.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.225+0000 7fa01dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:57.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.225+0000 7fa01dffb640 1 -- 172.21.15.62:0/376606368 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0100ab600 con 0x7fa0200a2900 2024-04-19T01:48:57.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.225+0000 7fa025892640 5 monclient: authenticate success, global_id 15324 2024-04-19T01:48:57.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.225+0000 7fa025892640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:48:57.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.225+0000 7fa01dffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:57.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.225+0000 7fa025892640 10 monclient: get_monmap_and_config success 2024-04-19T01:48:57.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.225+0000 7fa025892640 4 set_mon_vals no callback set 2024-04-19T01:48:57.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.225+0000 7fa01dffb640 1 -- 172.21.15.62:0/376606368 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0100ab9d0 con 0x7fa0200a2900 2024-04-19T01:48:57.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.225+0000 7fa01dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:57.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.225+0000 7fa01dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:48:57.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.225+0000 7fa01dffb640 10 monclient: dump: 2024-04-19T01:48:57.233 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:57.233 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:57.233 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:57.233 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:57.233 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:57.233 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:57.233 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:57.234 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:57.234 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:57.234 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:57.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.225+0000 7fa025892640 10 monclient: shutdown 2024-04-19T01:48:57.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.225+0000 7fa025892640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:57.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.225+0000 7fa025892640 1 -- 172.21.15.62:0/376606368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0200a2900 msgr2=0x7fa0200a46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:57.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.225+0000 7fa025892640 1 --2- 172.21.15.62:0/376606368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0200a2900 0x7fa0200a46d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fa020073940 tx=0x7fa01001c3e0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.225+0000 7fa025892640 1 -- 172.21.15.62:0/376606368 shutdown_connections 2024-04-19T01:48:57.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.225+0000 7fa025892640 1 --2- 172.21.15.62:0/376606368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0200a2900 0x7fa0200a46d0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.225+0000 7fa025892640 1 --2- 172.21.15.62:0/376606368 >> v2:172.21.15.71:3300/0 conn(0x7fa0200a3410 0x7fa0200a37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.225+0000 7fa025892640 1 -- 172.21.15.62:0/376606368 >> 172.21.15.62:0/376606368 conn(0x7fa0200aa2d0 msgr2=0x7fa0200a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:57.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.225+0000 7fa025892640 1 -- 172.21.15.62:0/376606368 shutdown_connections 2024-04-19T01:48:57.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.225+0000 7fa025892640 1 -- 172.21.15.62:0/376606368 wait complete. 2024-04-19T01:48:57.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa025892640 10 monclient: build_initial_monmap 2024-04-19T01:48:57.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa025892640 10 monclient: monmap: 2024-04-19T01:48:57.235 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:57.235 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:48:57.235 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:48:57.235 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:48:57.235 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:57.235 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:57.235 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:48:57.235 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:48:57.235 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:48:57.235 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:57.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa025892640 1 Processor -- start 2024-04-19T01:48:57.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa025892640 1 -- start start 2024-04-19T01:48:57.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa025892640 10 monclient: init 2024-04-19T01:48:57.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa025892640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:57.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa025892640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-19T01:48:57.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa025892640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0200a2900 0x7fa02013e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:57.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa025892640 10 monclient(hunting): picked mon.noname-a con 0x7fa0200a2900 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:48:57.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa025892640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa0200a2d50 0x7fa020125110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:57.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa025892640 10 monclient(hunting): picked mon.noname-b con 0x7fa0200a2d50 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:48:57.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa025892640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa0200a3410 0x7fa020125650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:57.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa025892640 10 monclient(hunting): picked mon.noname-c con 0x7fa0200a3410 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:57.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa025892640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:57.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa025892640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa020142380 con 0x7fa0200a2900 2024-04-19T01:48:57.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa025892640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:57.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa025892640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa020142200 con 0x7fa0200a2d50 2024-04-19T01:48:57.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa025892640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:57.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa025892640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fa020142500 con 0x7fa0200a3410 2024-04-19T01:48:57.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa025892640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:57.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa025892640 10 monclient(hunting): authenticate will time out at 1323.049133s 2024-04-19T01:48:57.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01effd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0200a2900 0x7fa02013e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:57.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01effd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0200a2900 0x7fa02013e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47104/0 (socket says 172.21.15.62:47104) 2024-04-19T01:48:57.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01effd640 1 -- 172.21.15.62:0/2695767669 learned_addr learned my addr 172.21.15.62:0/2695767669 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:57.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01effd640 10 monclient(hunting): get_auth_request con 0x7fa0200a2900 auth_method 0 2024-04-19T01:48:57.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:57.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01effd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:57.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:57.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01f7fe640 1 --2- 172.21.15.62:0/2695767669 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa0200a3410 0x7fa020125650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:57.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01f7fe640 10 monclient(hunting): get_auth_request con 0x7fa0200a3410 auth_method 0 2024-04-19T01:48:57.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:57.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:57.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:57.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:57.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:57.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:57.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:57.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:57.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:57.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01e7fc640 1 --2- 172.21.15.62:0/2695767669 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa0200a2d50 0x7fa020125110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:57.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01effd640 10 monclient(hunting): handle_auth_done global_id 15327 payload 824 2024-04-19T01:48:57.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01effd640 1 -- 172.21.15.62:0/2695767669 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa0200a3410 msgr2=0x7fa020125650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:57.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01effd640 1 --2- 172.21.15.62:0/2695767669 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa0200a3410 0x7fa020125650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01effd640 1 -- 172.21.15.62:0/2695767669 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa0200a2d50 msgr2=0x7fa020125110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:57.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01effd640 1 --2- 172.21.15.62:0/2695767669 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa0200a2d50 0x7fa020125110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01effd640 10 monclient: _finish_hunting 0 2024-04-19T01:48:57.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01effd640 1 monclient: found mon.noname-a 2024-04-19T01:48:57.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:57.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:48:57.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01effd640 1 -- 172.21.15.62:0/2695767669 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa020128d40 con 0x7fa0200a2900 2024-04-19T01:48:57.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01e7fc640 10 monclient: get_auth_request con 0x7fa0200a2d50 auth_method 0 2024-04-19T01:48:57.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01e7fc640 1 --2- 172.21.15.62:0/2695767669 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa0200a2d50 0x7fa020125110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:48:57.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01effd640 1 -- 172.21.15.62:0/2695767669 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0200a2900 msgr2=0x7fa02013e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:57.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.229+0000 7fa01effd640 1 --2- 172.21.15.62:0/2695767669 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0200a2900 0x7fa02013e690 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fa01400d800 tx=0x7fa01400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:57.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.233+0000 7fa007fff640 1 -- 172.21.15.62:0/2695767669 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01400b8d0 con 0x7fa0200a2900 2024-04-19T01:48:57.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.233+0000 7fa007fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:57.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.233+0000 7fa007fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:48:57.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.233+0000 7fa007fff640 10 monclient: dump: 2024-04-19T01:48:57.241 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:57.241 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:57.241 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:57.241 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:57.241 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:57.241 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:57.241 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:57.241 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:57.241 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:57.242 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:57.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.233+0000 7fa007fff640 10 monclient: _finish_auth 0 2024-04-19T01:48:57.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.233+0000 7fa007fff640 10 monclient: _check_auth_tickets 2024-04-19T01:48:57.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.233+0000 7fa007fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:57.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.233+0000 7fa007fff640 1 -- 172.21.15.62:0/2695767669 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa014019070 con 0x7fa0200a2900 2024-04-19T01:48:57.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.233+0000 7fa007fff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:57.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.233+0000 7fa007fff640 1 -- 172.21.15.62:0/2695767669 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01400bc70 con 0x7fa0200a2900 2024-04-19T01:48:57.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.233+0000 7fa025892640 5 monclient: authenticate success, global_id 15327 2024-04-19T01:48:57.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.233+0000 7fa025892640 10 monclient: _renew_subs 2024-04-19T01:48:57.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.233+0000 7fa025892640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:57.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.233+0000 7fa025892640 1 -- 172.21.15.62:0/2695767669 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa020129020 con 0x7fa0200a2900 2024-04-19T01:48:57.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.233+0000 7fa007fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:57.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.233+0000 7fa007fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:48:57.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.233+0000 7fa007fff640 10 monclient: dump: 2024-04-19T01:48:57.243 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:57.243 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:57.243 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:57.243 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:57.243 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:57.243 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:57.243 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:57.243 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:57.243 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:57.243 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:57.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.233+0000 7fa01d7fa640 4 set_mon_vals no callback set 2024-04-19T01:48:57.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.233+0000 7fa025892640 10 monclient: _renew_subs 2024-04-19T01:48:57.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.233+0000 7fa025892640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:57.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.233+0000 7fa025892640 1 -- 172.21.15.62:0/2695767669 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0200a3f80 con 0x7fa0200a2900 2024-04-19T01:48:57.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.233+0000 7fa025892640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:57.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.233+0000 7fa025892640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:57.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.233+0000 7fa025892640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:57.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.233+0000 7fa025892640 1 -- 172.21.15.62:0/2695767669 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0200692b0 con 0x7fa0200a2900 2024-04-19T01:48:57.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.233+0000 7fa007fff640 1 -- 172.21.15.62:0/2695767669 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fa01401e460 con 0x7fa0200a2900 2024-04-19T01:48:57.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.233+0000 7fa007fff640 1 --2- 172.21.15.62:0/2695767669 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa00c036420 0x7fa00c0388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:57.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.233+0000 7fa007fff640 1 -- 172.21.15.62:0/2695767669 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(325..325 src has 115..325) ==== 4346+0+0 (secure 0 0 0) 0x7fa014049830 con 0x7fa0200a2900 2024-04-19T01:48:57.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.233+0000 7fa01e7fc640 1 --2- 172.21.15.62:0/2695767669 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa00c036420 0x7fa00c0388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:57.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.233+0000 7fa01e7fc640 10 monclient: get_auth_request con 0x7fa00c036420 auth_method 0 2024-04-19T01:48:57.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.237+0000 7fa01e7fc640 1 --2- 172.21.15.62:0/2695767669 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa00c036420 0x7fa00c0388e0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa0200738e0 tx=0x7fa010008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:57.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.237+0000 7fa007fff640 1 -- 172.21.15.62:0/2695767669 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa01400be10 con 0x7fa0200a2900 2024-04-19T01:48:57.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.237+0000 7fa007fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:57.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.237+0000 7fa007fff640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:48:57.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.401+0000 7fa025892640 1 -- 172.21.15.62:0/2695767669 --> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7fa020126cc0 con 0x7fa00c036420 2024-04-19T01:48:57.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.401+0000 7fa007fff640 1 -- 172.21.15.62:0/2695767669 <== mgr.4108 v2:172.21.15.71:6824/2133533992 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fa020126cc0 con 0x7fa00c036420 2024-04-19T01:48:57.397 INFO:teuthology.orchestra.run.smithi062.stderr:instructed osd(s) 1 to deep-scrub 2024-04-19T01:48:57.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.401+0000 7fa025892640 1 -- 172.21.15.62:0/2695767669 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa00c036420 msgr2=0x7fa00c0388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:57.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.401+0000 7fa025892640 1 --2- 172.21.15.62:0/2695767669 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa00c036420 0x7fa00c0388e0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa0200738e0 tx=0x7fa010008040 comp rx=0 tx=0).stop 2024-04-19T01:48:57.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.401+0000 7fa025892640 10 monclient: shutdown 2024-04-19T01:48:57.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.401+0000 7fa025892640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:57.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.401+0000 7fa025892640 1 -- 172.21.15.62:0/2695767669 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0200a2900 msgr2=0x7fa02013e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:57.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.401+0000 7fa025892640 1 --2- 172.21.15.62:0/2695767669 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0200a2900 0x7fa02013e690 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fa01400d800 tx=0x7fa01400dcc0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.401+0000 7fa025892640 1 -- 172.21.15.62:0/2695767669 shutdown_connections 2024-04-19T01:48:57.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.401+0000 7fa025892640 1 --2- 172.21.15.62:0/2695767669 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fa00c036420 0x7fa00c0388e0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.401+0000 7fa025892640 1 --2- 172.21.15.62:0/2695767669 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fa0200a3410 0x7fa020125650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.401+0000 7fa025892640 1 --2- 172.21.15.62:0/2695767669 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa0200a2d50 0x7fa020125110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.401+0000 7fa025892640 1 --2- 172.21.15.62:0/2695767669 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0200a2900 0x7fa02013e690 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.401+0000 7fa025892640 1 -- 172.21.15.62:0/2695767669 >> 172.21.15.62:0/2695767669 conn(0x7fa0200aa2d0 msgr2=0x7fa0200a45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:57.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.401+0000 7fa025892640 1 -- 172.21.15.62:0/2695767669 shutdown_connections 2024-04-19T01:48:57.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.401+0000 7fa025892640 1 -- 172.21.15.62:0/2695767669 wait complete. 2024-04-19T01:48:57.417 INFO:tasks.ceph:Scrubbing osd.2 2024-04-19T01:48:57.417 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-19T01:48:57.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.549+0000 7ff0520cc640 10 monclient: get_monmap_and_config 2024-04-19T01:48:57.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.549+0000 7ff0520cc640 10 monclient: build_initial_monmap 2024-04-19T01:48:57.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.549+0000 7ff0520cc640 10 monclient: monmap: 2024-04-19T01:48:57.547 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:57.547 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:57.547 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:48:57.553160+0000 2024-04-19T01:48:57.547 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:48:57.553160+0000 2024-04-19T01:48:57.547 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:57.548 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:57.548 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:48:57.548 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:48:57.548 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:48:57.548 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:48:57.548 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:48:57.548 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:57.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff0520cc640 1 Processor -- start 2024-04-19T01:48:57.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff0520cc640 1 -- start start 2024-04-19T01:48:57.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff0520cc640 10 monclient: init 2024-04-19T01:48:57.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff0520cc640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:57.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff0520cc640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-04-19T01:48:57.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff0520cc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff04c0a10a0 0x7ff04c0a1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:57.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff0520cc640 10 monclient(hunting): picked mon.noname-b con 0x7ff04c0a10a0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:57.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff0520cc640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff04c0a19c0 0x7ff04c139820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:57.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff0520cc640 10 monclient(hunting): picked mon.noname-c con 0x7ff04c0a19c0 addr v2:172.21.15.71:3300/0 2024-04-19T01:48:57.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff0520cc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff04c139d60 0x7ff04c13c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:57.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff0520cc640 10 monclient(hunting): picked mon.noname-a con 0x7ff04c139d60 addr v2:172.21.15.62:3300/0 2024-04-19T01:48:57.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff0520cc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:57.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff0520cc640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7ff04c099150 con 0x7ff04c139d60 2024-04-19T01:48:57.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff0520cc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:57.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff0520cc640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7ff04c09dfc0 con 0x7ff04c0a19c0 2024-04-19T01:48:57.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff0520cc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:57.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff0520cc640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7ff04c09e580 con 0x7ff04c0a10a0 2024-04-19T01:48:57.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff0520cc640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:57.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff0520cc640 10 monclient(hunting): authenticate will time out at 1323.373059s 2024-04-19T01:48:57.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04bfff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff04c139d60 0x7ff04c13c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:57.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04b7fe640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff04c0a10a0 0x7ff04c0a1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:57.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04bfff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff04c139d60 0x7ff04c13c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47106/0 (socket says 172.21.15.62:47106) 2024-04-19T01:48:57.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04bfff640 1 -- 172.21.15.62:0/3957272339 learned_addr learned my addr 172.21.15.62:0/3957272339 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:57.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04bfff640 10 monclient(hunting): get_auth_request con 0x7ff04c139d60 auth_method 0 2024-04-19T01:48:57.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:57.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:57.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:57.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04b7fe640 10 monclient(hunting): get_auth_request con 0x7ff04c0a10a0 auth_method 0 2024-04-19T01:48:57.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:57.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:57.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:57.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:57.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:57.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:57.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:57.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:57.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:57.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04affd640 1 --2- 172.21.15.62:0/3957272339 >> v2:172.21.15.71:3300/0 conn(0x7ff04c0a19c0 0x7ff04c139820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:57.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04affd640 10 monclient(hunting): get_auth_request con 0x7ff04c0a19c0 auth_method 0 2024-04-19T01:48:57.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:57.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:57.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:57.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04b7fe640 10 monclient(hunting): handle_auth_done global_id 34160 payload 274 2024-04-19T01:48:57.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04b7fe640 1 -- 172.21.15.62:0/3957272339 >> v2:172.21.15.71:3300/0 conn(0x7ff04c0a19c0 msgr2=0x7ff04c139820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:57.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04b7fe640 1 --2- 172.21.15.62:0/3957272339 >> v2:172.21.15.71:3300/0 conn(0x7ff04c0a19c0 0x7ff04c139820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04b7fe640 1 -- 172.21.15.62:0/3957272339 >> v2:172.21.15.62:3300/0 conn(0x7ff04c139d60 msgr2=0x7ff04c13c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:57.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04b7fe640 1 --2- 172.21.15.62:0/3957272339 >> v2:172.21.15.62:3300/0 conn(0x7ff04c139d60 0x7ff04c13c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04b7fe640 10 monclient: _finish_hunting 0 2024-04-19T01:48:57.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04b7fe640 1 monclient: found mon.noname-b 2024-04-19T01:48:57.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:57.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:48:57.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04b7fe640 1 -- 172.21.15.62:0/3957272339 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff04c09f590 con 0x7ff04c0a10a0 2024-04-19T01:48:57.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.553+0000 7ff04bfff640 1 --2- 172.21.15.62:0/3957272339 >> v2:172.21.15.62:3300/0 conn(0x7ff04c139d60 0x7ff04c13c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:48:57.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff04b7fe640 1 -- 172.21.15.62:0/3957272339 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff04c0a10a0 msgr2=0x7ff04c0a1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:57.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff04b7fe640 1 --2- 172.21.15.62:0/3957272339 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff04c0a10a0 0x7ff04c0a1480 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff04c0a3210 tx=0x7ff03c01c340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=249eb4fa4c7ce720 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:57.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff04a7fc640 1 -- 172.21.15.62:0/3957272339 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03c0b0070 con 0x7ff04c0a10a0 2024-04-19T01:48:57.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff04a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:57.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff04a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:48:57.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff04a7fc640 10 monclient: dump: 2024-04-19T01:48:57.560 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:57.560 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:57.561 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:57.561 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:57.561 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:57.561 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:57.561 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:57.561 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:57.561 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:57.561 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:57.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff04a7fc640 10 monclient: _finish_auth 0 2024-04-19T01:48:57.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff04a7fc640 10 monclient: _check_auth_tickets 2024-04-19T01:48:57.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff04a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:57.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff04a7fc640 1 -- 172.21.15.62:0/3957272339 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff03c0ab600 con 0x7ff04c0a10a0 2024-04-19T01:48:57.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff0520cc640 5 monclient: authenticate success, global_id 34160 2024-04-19T01:48:57.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff0520cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:48:57.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff04a7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:57.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff0520cc640 10 monclient: get_monmap_and_config success 2024-04-19T01:48:57.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff04a7fc640 1 -- 172.21.15.62:0/3957272339 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03c0ab9d0 con 0x7ff04c0a10a0 2024-04-19T01:48:57.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff0520cc640 4 set_mon_vals no callback set 2024-04-19T01:48:57.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff04a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:57.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff04a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:48:57.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff04a7fc640 10 monclient: dump: 2024-04-19T01:48:57.562 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:57.562 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:57.562 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:57.562 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:57.562 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:57.562 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:57.562 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:57.563 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:57.563 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:57.563 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:57.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff0520cc640 10 monclient: shutdown 2024-04-19T01:48:57.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff0520cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:57.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff0520cc640 1 -- 172.21.15.62:0/3957272339 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff04c0a10a0 msgr2=0x7ff04c0a1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:57.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff0520cc640 1 --2- 172.21.15.62:0/3957272339 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff04c0a10a0 0x7ff04c0a1480 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff04c0a3210 tx=0x7ff03c01c340 comp rx=0 tx=0).stop 2024-04-19T01:48:57.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff0520cc640 1 -- 172.21.15.62:0/3957272339 shutdown_connections 2024-04-19T01:48:57.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff0520cc640 1 --2- 172.21.15.62:0/3957272339 >> v2:172.21.15.62:3300/0 conn(0x7ff04c139d60 0x7ff04c13c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff0520cc640 1 --2- 172.21.15.62:0/3957272339 >> v2:172.21.15.71:3300/0 conn(0x7ff04c0a19c0 0x7ff04c139820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff0520cc640 1 --2- 172.21.15.62:0/3957272339 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff04c0a10a0 0x7ff04c0a1480 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff0520cc640 1 -- 172.21.15.62:0/3957272339 >> 172.21.15.62:0/3957272339 conn(0x7ff04c0a2190 msgr2=0x7ff04c0a25a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:57.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff0520cc640 1 -- 172.21.15.62:0/3957272339 shutdown_connections 2024-04-19T01:48:57.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff0520cc640 1 -- 172.21.15.62:0/3957272339 wait complete. 2024-04-19T01:48:57.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff0520cc640 10 monclient: build_initial_monmap 2024-04-19T01:48:57.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff0520cc640 10 monclient: monmap: 2024-04-19T01:48:57.564 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:57.564 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:48:57.564 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:48:57.564 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:48:57.564 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:57.564 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:57.564 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:48:57.564 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:48:57.564 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:48:57.564 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:57.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff0520cc640 1 Processor -- start 2024-04-19T01:48:57.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff0520cc640 1 -- start start 2024-04-19T01:48:57.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff0520cc640 10 monclient: init 2024-04-19T01:48:57.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff0520cc640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:57.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff0520cc640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-19T01:48:57.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.557+0000 7ff0520cc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff04c0a10a0 0x7ff04c072c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:57.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff0520cc640 10 monclient(hunting): picked mon.noname-a con 0x7ff04c0a10a0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:48:57.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff0520cc640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff04c0a19c0 0x7ff04c099690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:57.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff0520cc640 10 monclient(hunting): picked mon.noname-c con 0x7ff04c0a19c0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:57.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff0520cc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff04c139d60 0x7ff04c099bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:57.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff0520cc640 10 monclient(hunting): picked mon.noname-b con 0x7ff04c139d60 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:48:57.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff0520cc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:57.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff0520cc640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff04c09f0f0 con 0x7ff04c0a10a0 2024-04-19T01:48:57.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff0520cc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:57.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff0520cc640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff04c09ef70 con 0x7ff04c139d60 2024-04-19T01:48:57.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff0520cc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:57.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff0520cc640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7ff04c09d6c0 con 0x7ff04c0a19c0 2024-04-19T01:48:57.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff0520cc640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:57.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff0520cc640 10 monclient(hunting): authenticate will time out at 1323.380057s 2024-04-19T01:48:57.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff04c0a10a0 0x7ff04c072c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:57.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04affd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff04c0a19c0 0x7ff04c099690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:57.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04affd640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff04c0a19c0 0x7ff04c099690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:40010/0 (socket says 172.21.15.62:40010) 2024-04-19T01:48:57.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04affd640 1 -- 172.21.15.62:0/1361295473 learned_addr learned my addr 172.21.15.62:0/1361295473 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:57.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04affd640 10 monclient(hunting): get_auth_request con 0x7ff04c0a19c0 auth_method 0 2024-04-19T01:48:57.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:57.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:57.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:57.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04b7fe640 10 monclient(hunting): get_auth_request con 0x7ff04c0a10a0 auth_method 0 2024-04-19T01:48:57.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:57.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:57.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:57.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:57.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:57.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:57.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:57.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:57.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:57.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04bfff640 1 --2- 172.21.15.62:0/1361295473 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff04c139d60 0x7ff04c099bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:57.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04bfff640 10 monclient(hunting): get_auth_request con 0x7ff04c139d60 auth_method 0 2024-04-19T01:48:57.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:57.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:57.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:57.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04b7fe640 10 monclient(hunting): handle_auth_done global_id 15333 payload 824 2024-04-19T01:48:57.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04b7fe640 1 -- 172.21.15.62:0/1361295473 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff04c0a19c0 msgr2=0x7ff04c099690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:57.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04b7fe640 1 --2- 172.21.15.62:0/1361295473 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff04c0a19c0 0x7ff04c099690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04b7fe640 1 -- 172.21.15.62:0/1361295473 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff04c139d60 msgr2=0x7ff04c099bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:57.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04b7fe640 1 --2- 172.21.15.62:0/1361295473 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff04c139d60 0x7ff04c099bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04b7fe640 10 monclient: _finish_hunting 0 2024-04-19T01:48:57.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04b7fe640 1 monclient: found mon.noname-a 2024-04-19T01:48:57.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:57.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:48:57.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04b7fe640 1 -- 172.21.15.62:0/1361295473 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff04c071730 con 0x7ff04c0a10a0 2024-04-19T01:48:57.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04affd640 1 --2- 172.21.15.62:0/1361295473 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ff04c0a19c0 0x7ff04c099690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:48:57.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04b7fe640 1 -- 172.21.15.62:0/1361295473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff04c0a10a0 msgr2=0x7ff04c072c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:57.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff04b7fe640 1 --2- 172.21.15.62:0/1361295473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff04c0a10a0 0x7ff04c072c90 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ff03c0ab000 tx=0x7ff03c01ab80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:57.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff033fff640 1 -- 172.21.15.62:0/1361295473 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03c0b0050 con 0x7ff04c0a10a0 2024-04-19T01:48:57.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff033fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:57.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff033fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:48:57.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff033fff640 10 monclient: dump: 2024-04-19T01:48:57.569 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:57.569 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:57.569 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:57.569 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:57.569 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:57.570 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:57.570 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:57.570 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:57.570 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:57.570 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:57.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff033fff640 10 monclient: _finish_auth 0 2024-04-19T01:48:57.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff033fff640 10 monclient: _check_auth_tickets 2024-04-19T01:48:57.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff033fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:57.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff033fff640 1 -- 172.21.15.62:0/1361295473 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff03c0b1040 con 0x7ff04c0a10a0 2024-04-19T01:48:57.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff0520cc640 5 monclient: authenticate success, global_id 15333 2024-04-19T01:48:57.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff0520cc640 10 monclient: _renew_subs 2024-04-19T01:48:57.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff0520cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:57.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff0520cc640 1 -- 172.21.15.62:0/1361295473 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff04c09f380 con 0x7ff04c0a10a0 2024-04-19T01:48:57.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff033fff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:57.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff033fff640 1 -- 172.21.15.62:0/1361295473 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03c0b5a40 con 0x7ff04c0a10a0 2024-04-19T01:48:57.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff033fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:57.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff033fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:48:57.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff033fff640 10 monclient: dump: 2024-04-19T01:48:57.571 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:57.571 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:57.572 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:57.572 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:57.572 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:57.572 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:57.572 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:57.572 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:57.572 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:57.572 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:57.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff0520cc640 10 monclient: _renew_subs 2024-04-19T01:48:57.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff0520cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:57.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff049ffb640 4 set_mon_vals no callback set 2024-04-19T01:48:57.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff0520cc640 1 -- 172.21.15.62:0/1361295473 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff04c0722b0 con 0x7ff04c0a10a0 2024-04-19T01:48:57.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff0520cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:57.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff0520cc640 1 -- 172.21.15.62:0/1361295473 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff04c04fcf0 con 0x7ff04c0a10a0 2024-04-19T01:48:57.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.561+0000 7ff033fff640 1 -- 172.21.15.62:0/1361295473 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7ff03c0be020 con 0x7ff04c0a10a0 2024-04-19T01:48:57.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.565+0000 7ff033fff640 1 --2- 172.21.15.62:0/1361295473 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7ff038036450 0x7ff038038910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:57.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.565+0000 7ff033fff640 1 -- 172.21.15.62:0/1361295473 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(325..325 src has 115..325) ==== 4346+0+0 (secure 0 0 0) 0x7ff03c0e2ff0 con 0x7ff04c0a10a0 2024-04-19T01:48:57.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.565+0000 7ff033fff640 1 --2- 172.21.15.62:0/1361295473 >> [v2:172.21.15.62:6801/1128256055,v1:172.21.15.62:6803/1128256055] conn(0x7ff03803c250 0x7ff03803e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:57.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.565+0000 7ff033fff640 1 -- 172.21.15.62:0/1361295473 --> [v2:172.21.15.62:6801/1128256055,v1:172.21.15.62:6803/1128256055] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff03803ed20 con 0x7ff03803c250 2024-04-19T01:48:57.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.565+0000 7ff033fff640 1 -- 172.21.15.62:0/1361295473 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=325) ==== 24+0+0 (secure 0 0 0) 0x7ff03c0e55c0 con 0x7ff04c0a10a0 2024-04-19T01:48:57.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.565+0000 7ff033fff640 10 monclient: handle_get_version_reply finishing 1 version 325 2024-04-19T01:48:57.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.565+0000 7ff04bfff640 1 --2- 172.21.15.62:0/1361295473 >> [v2:172.21.15.62:6801/1128256055,v1:172.21.15.62:6803/1128256055] conn(0x7ff03803c250 0x7ff03803e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:57.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.565+0000 7ff04affd640 1 --2- 172.21.15.62:0/1361295473 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7ff038036450 0x7ff038038910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:57.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.565+0000 7ff04bfff640 10 monclient: get_auth_request con 0x7ff03803c250 auth_method 0 2024-04-19T01:48:57.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.565+0000 7ff04affd640 10 monclient: get_auth_request con 0x7ff038036450 auth_method 0 2024-04-19T01:48:57.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.565+0000 7ff04affd640 1 --2- 172.21.15.62:0/1361295473 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7ff038036450 0x7ff038038910 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff04c0712b0 tx=0x7ff040008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:57.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.569+0000 7ff04bfff640 1 --2- 172.21.15.62:0/1361295473 >> [v2:172.21.15.62:6801/1128256055,v1:172.21.15.62:6803/1128256055] conn(0x7ff03803c250 0x7ff03803e670 crc :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:57.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.569+0000 7ff033fff640 1 -- 172.21.15.62:0/1361295473 <== osd.2 v2:172.21.15.62:6801/1128256055 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff03803ed20 con 0x7ff03803c250 2024-04-19T01:48:57.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.581+0000 7ff0520cc640 1 -- 172.21.15.62:0/1361295473 --> [v2:172.21.15.62:6801/1128256055,v1:172.21.15.62:6803/1128256055] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff04c09b640 con 0x7ff03803c250 2024-04-19T01:48:57.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.581+0000 7ff033fff640 1 -- 172.21.15.62:0/1361295473 <== osd.2 v2:172.21.15.62:6801/1128256055 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff04c09b640 con 0x7ff03803c250 2024-04-19T01:48:57.578 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-19T01:48:57.578 INFO:teuthology.orchestra.run.smithi062.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T01:48:57.578 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-19T01:48:57.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.581+0000 7ff0520cc640 1 -- 172.21.15.62:0/1361295473 >> [v2:172.21.15.62:6801/1128256055,v1:172.21.15.62:6803/1128256055] conn(0x7ff03803c250 msgr2=0x7ff03803e670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:57.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.581+0000 7ff0520cc640 1 --2- 172.21.15.62:0/1361295473 >> [v2:172.21.15.62:6801/1128256055,v1:172.21.15.62:6803/1128256055] conn(0x7ff03803c250 0x7ff03803e670 crc :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.581+0000 7ff0520cc640 1 -- 172.21.15.62:0/1361295473 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7ff038036450 msgr2=0x7ff038038910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:57.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.581+0000 7ff0520cc640 1 --2- 172.21.15.62:0/1361295473 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7ff038036450 0x7ff038038910 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff04c0712b0 tx=0x7ff040008040 comp rx=0 tx=0).stop 2024-04-19T01:48:57.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.581+0000 7ff0520cc640 10 monclient: shutdown 2024-04-19T01:48:57.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.581+0000 7ff0520cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:57.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.581+0000 7ff0520cc640 1 -- 172.21.15.62:0/1361295473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff04c0a10a0 msgr2=0x7ff04c072c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:57.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.581+0000 7ff0520cc640 1 --2- 172.21.15.62:0/1361295473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff04c0a10a0 0x7ff04c072c90 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ff03c0ab000 tx=0x7ff03c01ab80 comp rx=0 tx=0).stop 2024-04-19T01:48:57.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.581+0000 7ff04bfff640 1 -- 172.21.15.62:0/1361295473 reap_dead start 2024-04-19T01:48:57.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.581+0000 7ff0520cc640 1 -- 172.21.15.62:0/1361295473 shutdown_connections 2024-04-19T01:48:57.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.581+0000 7ff0520cc640 1 -- 172.21.15.62:0/1361295473 >> 172.21.15.62:0/1361295473 conn(0x7ff04c0a2190 msgr2=0x7ff04c136b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:57.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.581+0000 7ff0520cc640 1 -- 172.21.15.62:0/1361295473 shutdown_connections 2024-04-19T01:48:57.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.581+0000 7ff0520cc640 1 -- 172.21.15.62:0/1361295473 wait complete. 2024-04-19T01:48:57.593 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-19T01:48:57.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac28acf640 10 monclient: get_monmap_and_config 2024-04-19T01:48:57.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac28acf640 10 monclient: build_initial_monmap 2024-04-19T01:48:57.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac28acf640 10 monclient: monmap: 2024-04-19T01:48:57.705 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:57.705 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:57.705 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:48:57.711320+0000 2024-04-19T01:48:57.705 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:48:57.711320+0000 2024-04-19T01:48:57.706 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:57.706 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:57.706 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:48:57.706 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:48:57.706 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:48:57.706 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:48:57.706 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:48:57.706 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:57.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac28acf640 1 Processor -- start 2024-04-19T01:48:57.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac28acf640 1 -- start start 2024-04-19T01:48:57.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac28acf640 10 monclient: init 2024-04-19T01:48:57.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac28acf640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:57.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac28acf640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-04-19T01:48:57.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac28acf640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fac24137780 0x7fac24137b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:57.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac28acf640 10 monclient(hunting): picked mon.noname-b con 0x7fac24137780 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:57.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac28acf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fac241380a0 addr v1:172.21.15.71:6789/0 2024-04-19T01:48:57.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac28acf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fac240a60b0 addr v1:172.21.15.62:6789/0 2024-04-19T01:48:57.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac28acf640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac2409c2e0 con 0x7fac240a60b0 2024-04-19T01:48:57.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac28acf640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac240a53f0 con 0x7fac241380a0 2024-04-19T01:48:57.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac28acf640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:57.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac28acf640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fac2409bd20 con 0x7fac24137780 2024-04-19T01:48:57.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac28acf640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:57.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac28acf640 10 monclient(hunting): authenticate will time out at 1323.530821s 2024-04-19T01:48:57.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac2259c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fac24137780 0x7fac24137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:57.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac22d9d640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fac240a60b0 0x7fac240a4ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53254/0 (socket says 172.21.15.62:53254) 2024-04-19T01:48:57.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac22d9d640 1 -- 172.21.15.62:0/1016104676 learned_addr learned my addr 172.21.15.62:0/1016104676 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:48:57.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac2259c640 10 monclient(hunting): get_auth_request con 0x7fac24137780 auth_method 0 2024-04-19T01:48:57.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac2259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:57.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac2259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:57.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac2259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:57.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac2259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:57.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac2259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:57.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac2259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:57.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac2259c640 10 monclient(hunting): handle_auth_done global_id 34166 payload 274 2024-04-19T01:48:57.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac2259c640 1 -- 172.21.15.62:0/1016104676 >> v1:172.21.15.71:6789/0 conn(0x7fac241380a0 legacy=0x7fac240a45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:57.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac2259c640 1 -- 172.21.15.62:0/1016104676 >> v1:172.21.15.62:6789/0 conn(0x7fac240a60b0 legacy=0x7fac240a4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:57.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac2259c640 10 monclient: _finish_hunting 0 2024-04-19T01:48:57.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac2259c640 1 monclient: found mon.noname-b 2024-04-19T01:48:57.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac2259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:57.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac2259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:48:57.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.709+0000 7fac2259c640 1 -- 172.21.15.62:0/1016104676 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac2409c000 con 0x7fac24137780 2024-04-19T01:48:57.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac2259c640 1 -- 172.21.15.62:0/1016104676 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fac24137780 msgr2=0x7fac24137b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:57.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac2259c640 1 --2- 172.21.15.62:0/1016104676 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fac24137780 0x7fac24137b60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fac140097f0 tx=0x7fac1401c1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5990b55b70a3a16b server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:57.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac2159a640 1 -- 172.21.15.62:0/1016104676 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac140b0070 con 0x7fac24137780 2024-04-19T01:48:57.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac2159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:57.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac2159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:48:57.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac2159a640 10 monclient: dump: 2024-04-19T01:48:57.713 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:57.713 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:57.713 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:57.713 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:57.713 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:57.714 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:57.714 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:57.714 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:57.714 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:57.714 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:57.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac2159a640 10 monclient: _finish_auth 0 2024-04-19T01:48:57.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac2159a640 10 monclient: _check_auth_tickets 2024-04-19T01:48:57.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac2159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:57.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac2159a640 1 -- 172.21.15.62:0/1016104676 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac140ab600 con 0x7fac24137780 2024-04-19T01:48:57.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac2159a640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:57.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac2159a640 1 -- 172.21.15.62:0/1016104676 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac140ab9d0 con 0x7fac24137780 2024-04-19T01:48:57.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 5 monclient: authenticate success, global_id 34166 2024-04-19T01:48:57.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 10 monclient: get_monmap_and_config success 2024-04-19T01:48:57.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 4 set_mon_vals no callback set 2024-04-19T01:48:57.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac2159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:57.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac2159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:48:57.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac2159a640 10 monclient: dump: 2024-04-19T01:48:57.715 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:57.715 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:57.715 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:57.715 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:57.715 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:57.715 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:57.715 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:57.715 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:57.715 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:57.715 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:57.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 10 monclient: shutdown 2024-04-19T01:48:57.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:57.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 1 -- 172.21.15.62:0/1016104676 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fac24137780 msgr2=0x7fac24137b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:57.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 1 --2- 172.21.15.62:0/1016104676 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fac24137780 0x7fac24137b60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fac140097f0 tx=0x7fac1401c1c0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 1 -- 172.21.15.62:0/1016104676 shutdown_connections 2024-04-19T01:48:57.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 1 --2- 172.21.15.62:0/1016104676 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fac24137780 0x7fac24137b60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 1 -- 172.21.15.62:0/1016104676 >> 172.21.15.62:0/1016104676 conn(0x7fac240aa1d0 msgr2=0x7fac240a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:57.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 1 -- 172.21.15.62:0/1016104676 shutdown_connections 2024-04-19T01:48:57.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 1 -- 172.21.15.62:0/1016104676 wait complete. 2024-04-19T01:48:57.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 10 monclient: build_initial_monmap 2024-04-19T01:48:57.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 10 monclient: monmap: 2024-04-19T01:48:57.720 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:57.720 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:48:57.720 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:48:57.720 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:48:57.720 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:57.720 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:57.720 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:48:57.720 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:48:57.720 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:48:57.720 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:57.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 1 Processor -- start 2024-04-19T01:48:57.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 1 -- start start 2024-04-19T01:48:57.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 10 monclient: init 2024-04-19T01:48:57.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:57.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-19T01:48:57.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac240a60b0 0x7fac240a1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:57.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 10 monclient(hunting): picked mon.noname-b con 0x7fac240a60b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:48:57.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac24137780 0x7fac2409daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:57.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 10 monclient(hunting): picked mon.noname-a con 0x7fac24137780 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:48:57.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fac241380a0 0x7fac2409dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:57.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 10 monclient(hunting): picked mon.noname-c con 0x7fac241380a0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:57.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:57.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fac24143300 con 0x7fac24137780 2024-04-19T01:48:57.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:57.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fac24143180 con 0x7fac240a60b0 2024-04-19T01:48:57.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:57.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fac24143480 con 0x7fac241380a0 2024-04-19T01:48:57.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:57.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac28acf640 10 monclient(hunting): authenticate will time out at 1323.539449s 2024-04-19T01:48:57.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac21d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac24137780 0x7fac2409daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:57.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac22d9d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fac241380a0 0x7fac2409dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:57.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac21d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac24137780 0x7fac2409daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47130/0 (socket says 172.21.15.62:47130) 2024-04-19T01:48:57.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac21d9b640 1 -- 172.21.15.62:0/3989037086 learned_addr learned my addr 172.21.15.62:0/3989037086 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:57.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac21d9b640 10 monclient(hunting): get_auth_request con 0x7fac24137780 auth_method 0 2024-04-19T01:48:57.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac21d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:57.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac21d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:57.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac21d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:57.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac22d9d640 10 monclient(hunting): get_auth_request con 0x7fac241380a0 auth_method 0 2024-04-19T01:48:57.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac22d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:57.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac22d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:57.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac22d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:57.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac21d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:57.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac21d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:57.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac21d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:57.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac22d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:57.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac22d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:57.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac22d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:57.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac2259c640 1 --2- 172.21.15.62:0/3989037086 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac240a60b0 0x7fac240a1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:57.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac2259c640 10 monclient(hunting): get_auth_request con 0x7fac240a60b0 auth_method 0 2024-04-19T01:48:57.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac2259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:57.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac2259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:57.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac2259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:57.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac22d9d640 10 monclient(hunting): handle_auth_done global_id 34169 payload 824 2024-04-19T01:48:57.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac22d9d640 1 -- 172.21.15.62:0/3989037086 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac240a60b0 msgr2=0x7fac240a1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:57.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac22d9d640 1 --2- 172.21.15.62:0/3989037086 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac240a60b0 0x7fac240a1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.717+0000 7fac22d9d640 1 -- 172.21.15.62:0/3989037086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac24137780 msgr2=0x7fac2409daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:57.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac22d9d640 1 --2- 172.21.15.62:0/3989037086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac24137780 0x7fac2409daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac22d9d640 10 monclient: _finish_hunting 0 2024-04-19T01:48:57.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac22d9d640 1 monclient: found mon.noname-c 2024-04-19T01:48:57.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac22d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:57.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac22d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:48:57.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac22d9d640 1 -- 172.21.15.62:0/3989037086 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac240a1280 con 0x7fac241380a0 2024-04-19T01:48:57.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac21d9b640 1 --2- 172.21.15.62:0/3989037086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac24137780 0x7fac2409daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:48:57.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac22d9d640 1 -- 172.21.15.62:0/3989037086 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fac241380a0 msgr2=0x7fac2409dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:57.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac22d9d640 1 --2- 172.21.15.62:0/3989037086 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fac241380a0 0x7fac2409dfe0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fac1c00caf0 tx=0x7fac1c00cef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:57.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac12ffd640 1 -- 172.21.15.62:0/3989037086 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac1c00ade0 con 0x7fac241380a0 2024-04-19T01:48:57.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac12ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:57.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac12ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:48:57.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac12ffd640 10 monclient: dump: 2024-04-19T01:48:57.725 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:57.725 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:57.725 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:57.725 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:57.725 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:57.725 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:57.725 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:57.725 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:57.725 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:57.725 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:57.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac12ffd640 10 monclient: _finish_auth 0 2024-04-19T01:48:57.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac12ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:48:57.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:57.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac12ffd640 1 -- 172.21.15.62:0/3989037086 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac1c012400 con 0x7fac241380a0 2024-04-19T01:48:57.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac28acf640 5 monclient: authenticate success, global_id 34169 2024-04-19T01:48:57.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac28acf640 10 monclient: _renew_subs 2024-04-19T01:48:57.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac28acf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:48:57.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac28acf640 1 -- 172.21.15.62:0/3989037086 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac240a1560 con 0x7fac241380a0 2024-04-19T01:48:57.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac12ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:57.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac12ffd640 1 -- 172.21.15.62:0/3989037086 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac1c0127f0 con 0x7fac241380a0 2024-04-19T01:48:57.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac12ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:57.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac12ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:48:57.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac12ffd640 10 monclient: dump: 2024-04-19T01:48:57.726 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:57.726 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:57.726 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:57.726 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:57.726 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:57.726 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:57.726 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:57.727 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:57.727 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:57.727 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:57.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac20d99640 4 set_mon_vals no callback set 2024-04-19T01:48:57.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac28acf640 10 monclient: _renew_subs 2024-04-19T01:48:57.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac28acf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:48:57.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac28acf640 1 -- 172.21.15.62:0/3989037086 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fac2409e6f0 con 0x7fac241380a0 2024-04-19T01:48:57.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac28acf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:57.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac28acf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:57.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac28acf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:48:57.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac12ffd640 1 -- 172.21.15.62:0/3989037086 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fac1c012990 con 0x7fac241380a0 2024-04-19T01:48:57.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac28acf640 1 -- 172.21.15.62:0/3989037086 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac2409ff80 con 0x7fac241380a0 2024-04-19T01:48:57.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac12ffd640 1 --2- 172.21.15.62:0/3989037086 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fac04036420 0x7fac040388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:57.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.721+0000 7fac12ffd640 1 -- 172.21.15.62:0/3989037086 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(325..325 src has 115..325) ==== 4346+0+0 (secure 0 0 0) 0x7fac1c049900 con 0x7fac241380a0 2024-04-19T01:48:57.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.725+0000 7fac2259c640 1 --2- 172.21.15.62:0/3989037086 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fac04036420 0x7fac040388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:57.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.725+0000 7fac2259c640 10 monclient: get_auth_request con 0x7fac04036420 auth_method 0 2024-04-19T01:48:57.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.725+0000 7fac2259c640 1 --2- 172.21.15.62:0/3989037086 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fac04036420 0x7fac040388e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fac140ab000 tx=0x7fac14008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:57.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.729+0000 7fac12ffd640 1 -- 172.21.15.62:0/3989037086 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fac1c0484b0 con 0x7fac241380a0 2024-04-19T01:48:57.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.729+0000 7fac12ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:57.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.729+0000 7fac12ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:48:57.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.885+0000 7fac28acf640 1 -- 172.21.15.62:0/3989037086 --> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7fac240a3420 con 0x7fac04036420 2024-04-19T01:48:57.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.885+0000 7fac12ffd640 1 -- 172.21.15.62:0/3989037086 <== mgr.4108 v2:172.21.15.71:6824/2133533992 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fac140ab480 con 0x7fac04036420 2024-04-19T01:48:57.882 INFO:teuthology.orchestra.run.smithi062.stderr:instructed osd(s) 2 to deep-scrub 2024-04-19T01:48:57.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.885+0000 7fac28acf640 1 -- 172.21.15.62:0/3989037086 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fac04036420 msgr2=0x7fac040388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:57.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.889+0000 7fac28acf640 1 --2- 172.21.15.62:0/3989037086 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fac04036420 0x7fac040388e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fac140ab000 tx=0x7fac14008040 comp rx=0 tx=0).stop 2024-04-19T01:48:57.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.889+0000 7fac28acf640 10 monclient: shutdown 2024-04-19T01:48:57.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.889+0000 7fac28acf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:57.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.889+0000 7fac28acf640 1 -- 172.21.15.62:0/3989037086 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fac241380a0 msgr2=0x7fac2409dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:57.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.889+0000 7fac28acf640 1 --2- 172.21.15.62:0/3989037086 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fac241380a0 0x7fac2409dfe0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fac1c00caf0 tx=0x7fac1c00cef0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.889+0000 7fac28acf640 1 -- 172.21.15.62:0/3989037086 shutdown_connections 2024-04-19T01:48:57.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.889+0000 7fac28acf640 1 --2- 172.21.15.62:0/3989037086 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fac04036420 0x7fac040388e0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.889+0000 7fac28acf640 1 --2- 172.21.15.62:0/3989037086 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fac241380a0 0x7fac2409dfe0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.889+0000 7fac28acf640 1 --2- 172.21.15.62:0/3989037086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac24137780 0x7fac2409daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.889+0000 7fac28acf640 1 --2- 172.21.15.62:0/3989037086 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac240a60b0 0x7fac240a1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:57.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.889+0000 7fac28acf640 1 -- 172.21.15.62:0/3989037086 >> 172.21.15.62:0/3989037086 conn(0x7fac240aa1d0 msgr2=0x7fac24130520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:57.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.889+0000 7fac28acf640 1 -- 172.21.15.62:0/3989037086 shutdown_connections 2024-04-19T01:48:57.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:57.889+0000 7fac28acf640 1 -- 172.21.15.62:0/3989037086 wait complete. 2024-04-19T01:48:57.900 INFO:tasks.ceph:Scrubbing osd.3 2024-04-19T01:48:57.900 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-19T01:48:58.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.029+0000 7f88a5e31640 10 monclient: get_monmap_and_config 2024-04-19T01:48:58.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.029+0000 7f88a5e31640 10 monclient: build_initial_monmap 2024-04-19T01:48:58.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.029+0000 7f88a5e31640 10 monclient: monmap: 2024-04-19T01:48:58.027 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:58.027 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:58.027 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:48:58.033516+0000 2024-04-19T01:48:58.028 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:48:58.033516+0000 2024-04-19T01:48:58.028 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:58.028 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:58.028 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:48:58.028 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:48:58.028 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:48:58.028 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:48:58.028 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:48:58.028 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:58.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f88a5e31640 1 Processor -- start 2024-04-19T01:48:58.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f88a5e31640 1 -- start start 2024-04-19T01:48:58.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f88a5e31640 10 monclient: init 2024-04-19T01:48:58.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f88a5e31640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:58.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f88a5e31640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-04-19T01:48:58.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f88a5e31640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f88a01355f0 0x7f88a0072bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:58.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f88a5e31640 10 monclient(hunting): picked mon.noname-a con 0x7f88a01355f0 addr v2:172.21.15.62:3300/0 2024-04-19T01:48:58.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f88a5e31640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f88a01359d0 addr v1:172.21.15.71:6789/0 2024-04-19T01:48:58.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f88a5e31640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f88a00a6e00 addr v1:172.21.15.62:6789/0 2024-04-19T01:48:58.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f88a5e31640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88a0071820 con 0x7f88a00a6e00 2024-04-19T01:48:58.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f88a5e31640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88a0071b00 con 0x7f88a01359d0 2024-04-19T01:48:58.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f88a5e31640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:58.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f88a5e31640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f88a0070620 con 0x7f88a01355f0 2024-04-19T01:48:58.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f88a5e31640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:58.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f88a5e31640 10 monclient(hunting): authenticate will time out at 1323.853021s 2024-04-19T01:48:58.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f889f7fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f88a01355f0 0x7f88a0072bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:58.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f889ffff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f88a00a6e00 0x7f88a013d350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53260/0 (socket says 172.21.15.62:53260) 2024-04-19T01:48:58.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f889ffff640 1 -- 172.21.15.62:0/432223744 learned_addr learned my addr 172.21.15.62:0/432223744 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:48:58.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f889f7fe640 10 monclient(hunting): get_auth_request con 0x7f88a01355f0 auth_method 0 2024-04-19T01:48:58.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f889f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:58.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f889f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:58.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f889f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:58.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f889f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:58.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f889f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:58.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f889f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:58.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f889f7fe640 10 monclient(hunting): handle_auth_done global_id 15339 payload 274 2024-04-19T01:48:58.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f889f7fe640 1 -- 172.21.15.62:0/432223744 >> v1:172.21.15.71:6789/0 conn(0x7f88a01359d0 legacy=0x7f88a00a66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:58.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f889f7fe640 1 -- 172.21.15.62:0/432223744 >> v1:172.21.15.62:6789/0 conn(0x7f88a00a6e00 legacy=0x7f88a013d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:58.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f889f7fe640 10 monclient: _finish_hunting 0 2024-04-19T01:48:58.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f889f7fe640 1 monclient: found mon.noname-a 2024-04-19T01:48:58.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f889f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:58.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f889f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:48:58.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.033+0000 7f889f7fe640 1 -- 172.21.15.62:0/432223744 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a0071de0 con 0x7f88a01355f0 2024-04-19T01:48:58.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.037+0000 7f889f7fe640 1 -- 172.21.15.62:0/432223744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88a01355f0 msgr2=0x7f88a0072bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:58.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.037+0000 7f889f7fe640 1 --2- 172.21.15.62:0/432223744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88a01355f0 0x7f88a0072bf0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f88900097c0 tx=0x7f889001c180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34785d2242744c98 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:58.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.037+0000 7f889effd640 1 -- 172.21.15.62:0/432223744 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88900b1070 con 0x7f88a01355f0 2024-04-19T01:48:58.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.037+0000 7f889effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:58.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.037+0000 7f889effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:48:58.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.037+0000 7f889effd640 10 monclient: dump: 2024-04-19T01:48:58.035 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:58.035 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:58.035 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:58.035 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:58.035 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:58.035 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:58.036 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:58.036 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:58.036 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:58.036 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:58.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.037+0000 7f889effd640 10 monclient: _finish_auth 0 2024-04-19T01:48:58.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.037+0000 7f889effd640 10 monclient: _check_auth_tickets 2024-04-19T01:48:58.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.037+0000 7f889effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:58.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.037+0000 7f889effd640 1 -- 172.21.15.62:0/432223744 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88900ac440 con 0x7f88a01355f0 2024-04-19T01:48:58.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.037+0000 7f88a5e31640 5 monclient: authenticate success, global_id 15339 2024-04-19T01:48:58.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.037+0000 7f88a5e31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:48:58.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.037+0000 7f889effd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:58.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.037+0000 7f889effd640 1 -- 172.21.15.62:0/432223744 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88900ac810 con 0x7f88a01355f0 2024-04-19T01:48:58.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.037+0000 7f889effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:58.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.037+0000 7f889effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:48:58.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.037+0000 7f889effd640 10 monclient: dump: 2024-04-19T01:48:58.036 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:58.037 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:58.037 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:58.037 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:58.037 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:58.037 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:58.037 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:58.037 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:58.037 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:58.037 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:58.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.037+0000 7f88a5e31640 10 monclient: get_monmap_and_config success 2024-04-19T01:48:58.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.037+0000 7f88a5e31640 4 set_mon_vals no callback set 2024-04-19T01:48:58.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f88a5e31640 10 monclient: shutdown 2024-04-19T01:48:58.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f88a5e31640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:58.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f88a5e31640 1 -- 172.21.15.62:0/432223744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88a01355f0 msgr2=0x7f88a0072bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:58.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f88a5e31640 1 --2- 172.21.15.62:0/432223744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88a01355f0 0x7f88a0072bf0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f88900097c0 tx=0x7f889001c180 comp rx=0 tx=0).stop 2024-04-19T01:48:58.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f88a5e31640 1 -- 172.21.15.62:0/432223744 shutdown_connections 2024-04-19T01:48:58.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f88a5e31640 1 --2- 172.21.15.62:0/432223744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88a01355f0 0x7f88a0072bf0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f88a5e31640 1 -- 172.21.15.62:0/432223744 >> 172.21.15.62:0/432223744 conn(0x7f88a00a42e0 msgr2=0x7f88a00a46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:58.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f88a5e31640 1 -- 172.21.15.62:0/432223744 shutdown_connections 2024-04-19T01:48:58.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f88a5e31640 1 -- 172.21.15.62:0/432223744 wait complete. 2024-04-19T01:48:58.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f88a5e31640 10 monclient: build_initial_monmap 2024-04-19T01:48:58.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f88a5e31640 10 monclient: monmap: 2024-04-19T01:48:58.045 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:58.046 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:48:58.046 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:48:58.046 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:48:58.046 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:58.046 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:58.046 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:48:58.046 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:48:58.046 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:48:58.046 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:58.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f88a5e31640 1 Processor -- start 2024-04-19T01:48:58.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f88a5e31640 1 -- start start 2024-04-19T01:48:58.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f88a5e31640 10 monclient: init 2024-04-19T01:48:58.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f88a5e31640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:58.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f88a5e31640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-19T01:48:58.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f88a5e31640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88a00a6e00 0x7f88a01cda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:58.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f88a5e31640 10 monclient(hunting): picked mon.noname-a con 0x7f88a00a6e00 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:48:58.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f88a5e31640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88a01355f0 0x7f88a01cdf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:58.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f88a5e31640 10 monclient(hunting): picked mon.noname-c con 0x7f88a01355f0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:58.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f88a5e31640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88a01359d0 0x7f88a01ce490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:58.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f88a5e31640 10 monclient(hunting): picked mon.noname-b con 0x7f88a01359d0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:48:58.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f88a5e31640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:58.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f88a5e31640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f88a0141820 con 0x7f88a00a6e00 2024-04-19T01:48:58.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f88a5e31640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:58.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f88a5e31640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f88a01416a0 con 0x7f88a01359d0 2024-04-19T01:48:58.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f88a5e31640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:58.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f88a5e31640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f88a01419a0 con 0x7f88a01355f0 2024-04-19T01:48:58.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f88a5e31640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:58.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f88a5e31640 10 monclient(hunting): authenticate will time out at 1323.862279s 2024-04-19T01:48:58.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f889f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88a00a6e00 0x7f88a01cda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:58.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f8897fff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88a01355f0 0x7f88a01cdf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:58.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f889f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88a00a6e00 0x7f88a01cda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47152/0 (socket says 172.21.15.62:47152) 2024-04-19T01:48:58.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f889f7fe640 1 -- 172.21.15.62:0/4229018462 learned_addr learned my addr 172.21.15.62:0/4229018462 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:58.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f8897fff640 10 monclient(hunting): get_auth_request con 0x7f88a01355f0 auth_method 0 2024-04-19T01:48:58.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f8897fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:58.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f8897fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:58.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f8897fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:58.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f889f7fe640 10 monclient(hunting): get_auth_request con 0x7f88a00a6e00 auth_method 0 2024-04-19T01:48:58.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f889f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:58.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f889f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:58.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f889f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:58.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f8897fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:58.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f8897fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:58.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f8897fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:58.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f889f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:58.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f889f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:58.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f889f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:58.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f889ffff640 1 --2- 172.21.15.62:0/4229018462 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88a01359d0 0x7f88a01ce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:58.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f889ffff640 10 monclient(hunting): get_auth_request con 0x7f88a01359d0 auth_method 0 2024-04-19T01:48:58.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f889ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:58.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f889ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:58.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f889ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:58.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f8897fff640 10 monclient(hunting): handle_auth_done global_id 34172 payload 824 2024-04-19T01:48:58.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f8897fff640 1 -- 172.21.15.62:0/4229018462 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88a01359d0 msgr2=0x7f88a01ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:58.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f8897fff640 1 --2- 172.21.15.62:0/4229018462 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f88a01359d0 0x7f88a01ce490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f8897fff640 1 -- 172.21.15.62:0/4229018462 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88a00a6e00 msgr2=0x7f88a01cda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:58.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f8897fff640 1 --2- 172.21.15.62:0/4229018462 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88a00a6e00 0x7f88a01cda10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f8897fff640 10 monclient: _finish_hunting 0 2024-04-19T01:48:58.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f8897fff640 1 monclient: found mon.noname-c 2024-04-19T01:48:58.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f8897fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:58.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f8897fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:48:58.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.041+0000 7f8897fff640 1 -- 172.21.15.62:0/4229018462 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a0131b10 con 0x7f88a01355f0 2024-04-19T01:48:58.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f8897fff640 1 -- 172.21.15.62:0/4229018462 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88a01355f0 msgr2=0x7f88a01cdf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:58.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f8897fff640 1 --2- 172.21.15.62:0/4229018462 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88a01355f0 0x7f88a01cdf50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f888800c3e0 tx=0x7f888800c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:58.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f889cff9640 1 -- 172.21.15.62:0/4229018462 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88880119a0 con 0x7f88a01355f0 2024-04-19T01:48:58.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f889cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:58.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f889cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:48:58.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f889cff9640 10 monclient: dump: 2024-04-19T01:48:58.051 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:58.051 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:58.051 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:58.051 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:58.051 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:58.051 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:58.051 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:58.051 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:58.051 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:58.051 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:58.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f889cff9640 10 monclient: _finish_auth 0 2024-04-19T01:48:58.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f889cff9640 10 monclient: _check_auth_tickets 2024-04-19T01:48:58.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f889cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:58.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f889cff9640 1 -- 172.21.15.62:0/4229018462 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8888017070 con 0x7f88a01355f0 2024-04-19T01:48:58.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f889cff9640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:58.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f889cff9640 1 -- 172.21.15.62:0/4229018462 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8888011d80 con 0x7f88a01355f0 2024-04-19T01:48:58.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f889cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:58.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f889cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:48:58.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f889cff9640 10 monclient: dump: 2024-04-19T01:48:58.052 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:58.052 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:58.052 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:58.052 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:58.052 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:58.052 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:58.052 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:58.052 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:58.052 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:58.052 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:58.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f88a5e31640 5 monclient: authenticate success, global_id 34172 2024-04-19T01:48:58.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f889dffb640 4 set_mon_vals no callback set 2024-04-19T01:48:58.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f88a5e31640 10 monclient: _renew_subs 2024-04-19T01:48:58.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f88a5e31640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:48:58.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f88a5e31640 1 -- 172.21.15.62:0/4229018462 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88a009ff90 con 0x7f88a01355f0 2024-04-19T01:48:58.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f88a5e31640 10 monclient: _renew_subs 2024-04-19T01:48:58.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f88a5e31640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:48:58.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f88a5e31640 1 -- 172.21.15.62:0/4229018462 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f88a009e330 con 0x7f88a01355f0 2024-04-19T01:48:58.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f88a5e31640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:48:58.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f88a5e31640 1 -- 172.21.15.62:0/4229018462 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f88a013e860 con 0x7f88a01355f0 2024-04-19T01:48:58.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f889cff9640 1 -- 172.21.15.62:0/4229018462 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f888801c460 con 0x7f88a01355f0 2024-04-19T01:48:58.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f889cff9640 1 --2- 172.21.15.62:0/4229018462 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f8880036470 0x7f8880038930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:58.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f889cff9640 1 -- 172.21.15.62:0/4229018462 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(325..325 src has 115..325) ==== 4346+0+0 (secure 0 0 0) 0x7f8888049130 con 0x7f88a01355f0 2024-04-19T01:48:58.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f889cff9640 1 --2- 172.21.15.62:0/4229018462 >> [v2:172.21.15.71:6816/2632024688,v1:172.21.15.71:6817/2632024688] conn(0x7f888003c2f0 0x7f888003e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:58.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f889cff9640 1 -- 172.21.15.62:0/4229018462 --> [v2:172.21.15.71:6816/2632024688,v1:172.21.15.71:6817/2632024688] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f888801ee70 con 0x7f888003c2f0 2024-04-19T01:48:58.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f889cff9640 1 -- 172.21.15.62:0/4229018462 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_get_version_reply(handle=1 version=325) ==== 24+0+0 (secure 0 0 0) 0x7f8888050050 con 0x7f88a01355f0 2024-04-19T01:48:58.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f889cff9640 10 monclient: handle_get_version_reply finishing 1 version 325 2024-04-19T01:48:58.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f889f7fe640 1 --2- 172.21.15.62:0/4229018462 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f8880036470 0x7f8880038930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:58.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f889ffff640 1 --2- 172.21.15.62:0/4229018462 >> [v2:172.21.15.71:6816/2632024688,v1:172.21.15.71:6817/2632024688] conn(0x7f888003c2f0 0x7f888003e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:58.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f889f7fe640 10 monclient: get_auth_request con 0x7f8880036470 auth_method 0 2024-04-19T01:48:58.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.045+0000 7f889ffff640 10 monclient: get_auth_request con 0x7f888003c2f0 auth_method 0 2024-04-19T01:48:58.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.049+0000 7f889f7fe640 1 --2- 172.21.15.62:0/4229018462 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f8880036470 0x7f8880038930 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f88900097c0 tx=0x7f88900a6070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:58.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.053+0000 7f889ffff640 1 --2- 172.21.15.62:0/4229018462 >> [v2:172.21.15.71:6816/2632024688,v1:172.21.15.71:6817/2632024688] conn(0x7f888003c2f0 0x7f888003e710 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:58.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.053+0000 7f889cff9640 1 -- 172.21.15.62:0/4229018462 <== osd.3 v2:172.21.15.71:6816/2632024688 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f888801ee70 con 0x7f888003c2f0 2024-04-19T01:48:58.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.065+0000 7f88a5e31640 1 -- 172.21.15.62:0/4229018462 --> [v2:172.21.15.71:6816/2632024688,v1:172.21.15.71:6817/2632024688] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f88a01302e0 con 0x7f888003c2f0 2024-04-19T01:48:58.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.065+0000 7f889cff9640 1 -- 172.21.15.62:0/4229018462 <== osd.3 v2:172.21.15.71:6816/2632024688 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f88a01302e0 con 0x7f888003c2f0 2024-04-19T01:48:58.064 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-19T01:48:58.064 INFO:teuthology.orchestra.run.smithi062.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T01:48:58.064 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-19T01:48:58.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.069+0000 7f88a5e31640 1 -- 172.21.15.62:0/4229018462 >> [v2:172.21.15.71:6816/2632024688,v1:172.21.15.71:6817/2632024688] conn(0x7f888003c2f0 msgr2=0x7f888003e710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:58.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.069+0000 7f88a5e31640 1 --2- 172.21.15.62:0/4229018462 >> [v2:172.21.15.71:6816/2632024688,v1:172.21.15.71:6817/2632024688] conn(0x7f888003c2f0 0x7f888003e710 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.069+0000 7f88a5e31640 1 -- 172.21.15.62:0/4229018462 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f8880036470 msgr2=0x7f8880038930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:58.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.069+0000 7f88a5e31640 1 --2- 172.21.15.62:0/4229018462 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f8880036470 0x7f8880038930 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f88900097c0 tx=0x7f88900a6070 comp rx=0 tx=0).stop 2024-04-19T01:48:58.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.069+0000 7f88a5e31640 10 monclient: shutdown 2024-04-19T01:48:58.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.069+0000 7f88a5e31640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:58.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.069+0000 7f88a5e31640 1 -- 172.21.15.62:0/4229018462 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88a01355f0 msgr2=0x7f88a01cdf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:58.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.069+0000 7f88a5e31640 1 --2- 172.21.15.62:0/4229018462 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f88a01355f0 0x7f88a01cdf50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f888800c3e0 tx=0x7f888800c8a0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.069+0000 7f889ffff640 1 -- 172.21.15.62:0/4229018462 reap_dead start 2024-04-19T01:48:58.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.069+0000 7f88a5e31640 1 -- 172.21.15.62:0/4229018462 shutdown_connections 2024-04-19T01:48:58.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.069+0000 7f88a5e31640 1 -- 172.21.15.62:0/4229018462 >> 172.21.15.62:0/4229018462 conn(0x7f88a00a42e0 msgr2=0x7f88a012f980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:58.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.069+0000 7f88a5e31640 1 -- 172.21.15.62:0/4229018462 shutdown_connections 2024-04-19T01:48:58.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.069+0000 7f88a5e31640 1 -- 172.21.15.62:0/4229018462 wait complete. 2024-04-19T01:48:58.083 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-19T01:48:58.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.213+0000 7ffa14284640 10 monclient: get_monmap_and_config 2024-04-19T01:48:58.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.213+0000 7ffa14284640 10 monclient: build_initial_monmap 2024-04-19T01:48:58.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.213+0000 7ffa14284640 10 monclient: monmap: 2024-04-19T01:48:58.211 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:58.211 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:58.211 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:48:58.216768+0000 2024-04-19T01:48:58.211 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:48:58.216768+0000 2024-04-19T01:48:58.211 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:58.211 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:58.211 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:48:58.211 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:48:58.211 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:48:58.211 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:48:58.212 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:48:58.212 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:58.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.213+0000 7ffa14284640 1 Processor -- start 2024-04-19T01:48:58.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.213+0000 7ffa14284640 1 -- start start 2024-04-19T01:48:58.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.213+0000 7ffa14284640 10 monclient: init 2024-04-19T01:48:58.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa14284640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:58.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa14284640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-04-19T01:48:58.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa14284640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ffa0c0a2ef0 0x7ffa0c1379a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:58.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa14284640 10 monclient(hunting): picked mon.noname-c con 0x7ffa0c0a2ef0 addr v2:172.21.15.71:3300/0 2024-04-19T01:48:58.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa14284640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ffa0c137ee0 0x7ffa0c0a47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:58.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa14284640 10 monclient(hunting): picked mon.noname-a con 0x7ffa0c137ee0 addr v2:172.21.15.62:3300/0 2024-04-19T01:48:58.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa14284640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffa0c0a4d20 addr v1:172.21.15.71:6789/0 2024-04-19T01:48:58.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa14284640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa0c137550 con 0x7ffa0c0a4d20 2024-04-19T01:48:58.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa14284640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:58.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa14284640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7ffa0c09bd90 con 0x7ffa0c137ee0 2024-04-19T01:48:58.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa14284640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:58.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa14284640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7ffa0c09c070 con 0x7ffa0c0a2ef0 2024-04-19T01:48:58.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa14284640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:58.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa14284640 10 monclient(hunting): authenticate will time out at 1324.036274s 2024-04-19T01:48:58.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa1181f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ffa0c137ee0 0x7ffa0c0a47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:58.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa1181f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ffa0c137ee0 0x7ffa0c0a47e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47160/0 (socket says 172.21.15.62:47160) 2024-04-19T01:48:58.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa1181f640 1 -- 172.21.15.62:0/2922338801 learned_addr learned my addr 172.21.15.62:0/2922338801 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:58.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa1181f640 10 monclient(hunting): get_auth_request con 0x7ffa0c137ee0 auth_method 0 2024-04-19T01:48:58.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa1181f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:58.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa1181f640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:58.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa1181f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:58.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa1181f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:58.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa1181f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:58.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa1181f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:58.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa12020640 1 --2- 172.21.15.62:0/2922338801 >> v2:172.21.15.71:3300/0 conn(0x7ffa0c0a2ef0 0x7ffa0c1379a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:58.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa12020640 10 monclient(hunting): get_auth_request con 0x7ffa0c0a2ef0 auth_method 0 2024-04-19T01:48:58.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa12020640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:58.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa12020640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:58.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa12020640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:58.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa1181f640 10 monclient(hunting): handle_auth_done global_id 15345 payload 274 2024-04-19T01:48:58.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa1181f640 1 -- 172.21.15.62:0/2922338801 >> v2:172.21.15.71:3300/0 conn(0x7ffa0c0a2ef0 msgr2=0x7ffa0c1379a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:58.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa1181f640 1 --2- 172.21.15.62:0/2922338801 >> v2:172.21.15.71:3300/0 conn(0x7ffa0c0a2ef0 0x7ffa0c1379a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa1181f640 1 -- 172.21.15.62:0/2922338801 >> v1:172.21.15.71:6789/0 conn(0x7ffa0c0a4d20 legacy=0x7ffa0c13e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:58.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa1181f640 10 monclient: _finish_hunting 0 2024-04-19T01:48:58.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa1181f640 1 monclient: found mon.noname-a 2024-04-19T01:48:58.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa1181f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:58.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa1181f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:48:58.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.217+0000 7ffa1181f640 1 -- 172.21.15.62:0/2922338801 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa0c09c350 con 0x7ffa0c137ee0 2024-04-19T01:48:58.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa1181f640 1 -- 172.21.15.62:0/2922338801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa0c137ee0 msgr2=0x7ffa0c0a47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:58.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa1181f640 1 --2- 172.21.15.62:0/2922338801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa0c137ee0 0x7ffa0c0a47e0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c072360 tx=0x7ff9fc01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81bb9bef8108f259 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:58.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa1101e640 1 -- 172.21.15.62:0/2922338801 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc0b0070 con 0x7ffa0c137ee0 2024-04-19T01:48:58.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa1101e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:58.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa1101e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:48:58.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa1101e640 10 monclient: dump: 2024-04-19T01:48:58.225 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:58.225 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:58.225 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:58.225 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:58.225 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:58.225 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:58.225 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:58.225 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:58.225 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:58.225 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:58.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa1101e640 10 monclient: _finish_auth 0 2024-04-19T01:48:58.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa1101e640 10 monclient: _check_auth_tickets 2024-04-19T01:48:58.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa1101e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:58.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa1101e640 1 -- 172.21.15.62:0/2922338801 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9fc0ab600 con 0x7ffa0c137ee0 2024-04-19T01:48:58.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa14284640 5 monclient: authenticate success, global_id 15345 2024-04-19T01:48:58.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa14284640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:48:58.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa1101e640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:58.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa14284640 10 monclient: get_monmap_and_config success 2024-04-19T01:48:58.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa14284640 4 set_mon_vals no callback set 2024-04-19T01:48:58.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa1101e640 1 -- 172.21.15.62:0/2922338801 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc0ab9d0 con 0x7ffa0c137ee0 2024-04-19T01:48:58.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa1101e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:58.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa1101e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:48:58.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa1101e640 10 monclient: dump: 2024-04-19T01:48:58.226 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:58.226 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:58.226 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:58.226 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:58.227 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:58.227 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:58.227 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:58.227 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:58.227 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:58.227 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:58.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa14284640 10 monclient: shutdown 2024-04-19T01:48:58.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa14284640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:58.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa14284640 1 -- 172.21.15.62:0/2922338801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa0c137ee0 msgr2=0x7ffa0c0a47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:58.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa14284640 1 --2- 172.21.15.62:0/2922338801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa0c137ee0 0x7ffa0c0a47e0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c072360 tx=0x7ff9fc01c470 comp rx=0 tx=0).stop 2024-04-19T01:48:58.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa14284640 1 -- 172.21.15.62:0/2922338801 shutdown_connections 2024-04-19T01:48:58.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa14284640 1 --2- 172.21.15.62:0/2922338801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa0c137ee0 0x7ffa0c0a47e0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa14284640 1 --2- 172.21.15.62:0/2922338801 >> v2:172.21.15.71:3300/0 conn(0x7ffa0c0a2ef0 0x7ffa0c1379a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa14284640 1 -- 172.21.15.62:0/2922338801 >> 172.21.15.62:0/2922338801 conn(0x7ffa0c0a23d0 msgr2=0x7ffa0c0a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:58.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa14284640 1 -- 172.21.15.62:0/2922338801 shutdown_connections 2024-04-19T01:48:58.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa14284640 1 -- 172.21.15.62:0/2922338801 wait complete. 2024-04-19T01:48:58.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa14284640 10 monclient: build_initial_monmap 2024-04-19T01:48:58.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa14284640 10 monclient: monmap: 2024-04-19T01:48:58.228 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:58.228 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:48:58.228 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:48:58.228 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:48:58.228 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:58.228 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:58.228 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:48:58.228 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:48:58.228 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:48:58.228 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:58.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa14284640 1 Processor -- start 2024-04-19T01:48:58.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa14284640 1 -- start start 2024-04-19T01:48:58.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.221+0000 7ffa14284640 10 monclient: init 2024-04-19T01:48:58.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa14284640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:58.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa14284640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-19T01:48:58.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa14284640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa0c0a2ef0 0x7ffa0c1cf970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:58.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa14284640 10 monclient(hunting): picked mon.noname-a con 0x7ffa0c0a2ef0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:48:58.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa14284640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffa0c0a4d20 0x7ffa0c1331f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:58.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa14284640 10 monclient(hunting): picked mon.noname-c con 0x7ffa0c0a4d20 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:58.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa14284640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa0c137ee0 0x7ffa0c13c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:58.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa14284640 10 monclient(hunting): picked mon.noname-b con 0x7ffa0c137ee0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:48:58.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa14284640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:58.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa14284640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0c1424d0 con 0x7ffa0c0a2ef0 2024-04-19T01:48:58.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa14284640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:58.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa14284640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0c072520 con 0x7ffa0c137ee0 2024-04-19T01:48:58.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa14284640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:58.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa14284640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7ffa0c142650 con 0x7ffa0c0a4d20 2024-04-19T01:48:58.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa14284640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:58.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa14284640 10 monclient(hunting): authenticate will time out at 1324.044302s 2024-04-19T01:48:58.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa12020640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa0c0a2ef0 0x7ffa0c1cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:58.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa12020640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa0c0a2ef0 0x7ffa0c1cf970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47168/0 (socket says 172.21.15.62:47168) 2024-04-19T01:48:58.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa12020640 1 -- 172.21.15.62:0/1581834028 learned_addr learned my addr 172.21.15.62:0/1581834028 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:58.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa12020640 10 monclient(hunting): get_auth_request con 0x7ffa0c0a2ef0 auth_method 0 2024-04-19T01:48:58.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa12020640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:58.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa12020640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:58.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa12020640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:58.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa1181f640 1 --2- 172.21.15.62:0/1581834028 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffa0c0a4d20 0x7ffa0c1331f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:58.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa1181f640 10 monclient(hunting): get_auth_request con 0x7ffa0c0a4d20 auth_method 0 2024-04-19T01:48:58.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa1181f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:58.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa1181f640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:58.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa1181f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:58.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa12020640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:58.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa12020640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:58.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa12020640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:58.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa1181f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:58.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa1181f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:58.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa1181f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:58.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa12821640 1 --2- 172.21.15.62:0/1581834028 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa0c137ee0 0x7ffa0c13c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:58.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa12821640 10 monclient(hunting): get_auth_request con 0x7ffa0c137ee0 auth_method 0 2024-04-19T01:48:58.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa12821640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:58.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa12821640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:58.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa12821640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:58.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa12020640 10 monclient(hunting): handle_auth_done global_id 15348 payload 824 2024-04-19T01:48:58.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa12020640 1 -- 172.21.15.62:0/1581834028 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffa0c0a4d20 msgr2=0x7ffa0c1331f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:58.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa12020640 1 --2- 172.21.15.62:0/1581834028 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffa0c0a4d20 0x7ffa0c1331f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa12020640 1 -- 172.21.15.62:0/1581834028 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa0c137ee0 msgr2=0x7ffa0c13c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:58.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa12020640 1 --2- 172.21.15.62:0/1581834028 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa0c137ee0 0x7ffa0c13c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa12020640 10 monclient: _finish_hunting 0 2024-04-19T01:48:58.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa12020640 1 monclient: found mon.noname-a 2024-04-19T01:48:58.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa12020640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:58.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa12020640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:48:58.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa12020640 1 -- 172.21.15.62:0/1581834028 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa0c11c380 con 0x7ffa0c0a2ef0 2024-04-19T01:48:58.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa12020640 1 -- 172.21.15.62:0/1581834028 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa0c0a2ef0 msgr2=0x7ffa0c1cf970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:58.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa12020640 1 --2- 172.21.15.62:0/1581834028 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa0c0a2ef0 0x7ffa0c1cf970 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ffa0800d5f0 tx=0x7ffa0800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:58.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa02ffd640 1 -- 172.21.15.62:0/1581834028 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa08012800 con 0x7ffa0c0a2ef0 2024-04-19T01:48:58.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:58.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa02ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:48:58.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa02ffd640 10 monclient: dump: 2024-04-19T01:48:58.232 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:58.232 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:58.232 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:58.232 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:58.232 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:58.233 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:58.233 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:58.233 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:58.233 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:58.233 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:58.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa02ffd640 10 monclient: _finish_auth 0 2024-04-19T01:48:58.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa02ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:48:58.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:58.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa02ffd640 1 -- 172.21.15.62:0/1581834028 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa08019070 con 0x7ffa0c0a2ef0 2024-04-19T01:48:58.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa14284640 5 monclient: authenticate success, global_id 15348 2024-04-19T01:48:58.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa14284640 10 monclient: _renew_subs 2024-04-19T01:48:58.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa14284640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:58.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa14284640 1 -- 172.21.15.62:0/1581834028 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa0c11d2e0 con 0x7ffa0c0a2ef0 2024-04-19T01:48:58.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa02ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:58.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa02ffd640 1 -- 172.21.15.62:0/1581834028 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa08012bc0 con 0x7ffa0c0a2ef0 2024-04-19T01:48:58.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:58.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa02ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:48:58.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa1081d640 4 set_mon_vals no callback set 2024-04-19T01:48:58.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa02ffd640 10 monclient: dump: 2024-04-19T01:48:58.234 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:58.234 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:58.234 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:58.234 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:58.234 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:58.234 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:58.234 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:58.234 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:58.234 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:58.234 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:58.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa14284640 10 monclient: _renew_subs 2024-04-19T01:48:58.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa14284640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:58.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.225+0000 7ffa14284640 1 -- 172.21.15.62:0/1581834028 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa0c1d07f0 con 0x7ffa0c0a2ef0 2024-04-19T01:48:58.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.229+0000 7ffa14284640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:58.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.229+0000 7ffa14284640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:58.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.229+0000 7ffa14284640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:58.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.229+0000 7ffa14284640 1 -- 172.21.15.62:0/1581834028 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa0c0692b0 con 0x7ffa0c0a2ef0 2024-04-19T01:48:58.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.229+0000 7ffa02ffd640 1 -- 172.21.15.62:0/1581834028 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7ffa0801e460 con 0x7ffa0c0a2ef0 2024-04-19T01:48:58.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.229+0000 7ffa02ffd640 1 --2- 172.21.15.62:0/1581834028 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7ff9f0036420 0x7ff9f00388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:58.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.229+0000 7ffa02ffd640 1 -- 172.21.15.62:0/1581834028 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(325..325 src has 115..325) ==== 4346+0+0 (secure 0 0 0) 0x7ffa08049750 con 0x7ffa0c0a2ef0 2024-04-19T01:48:58.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.229+0000 7ffa1181f640 1 --2- 172.21.15.62:0/1581834028 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7ff9f0036420 0x7ff9f00388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:58.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.229+0000 7ffa1181f640 10 monclient: get_auth_request con 0x7ff9f0036420 auth_method 0 2024-04-19T01:48:58.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.229+0000 7ffa1181f640 1 --2- 172.21.15.62:0/1581834028 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7ff9f0036420 0x7ff9f00388e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc0ab000 tx=0x7ff9fc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:58.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.233+0000 7ffa02ffd640 1 -- 172.21.15.62:0/1581834028 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ffa0801d580 con 0x7ffa0c0a2ef0 2024-04-19T01:48:58.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.233+0000 7ffa02ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:58.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.233+0000 7ffa02ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:48:58.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.393+0000 7ffa14284640 1 -- 172.21.15.62:0/1581834028 --> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7ffa0c13e020 con 0x7ff9f0036420 2024-04-19T01:48:58.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.397+0000 7ffa02ffd640 1 -- 172.21.15.62:0/1581834028 <== mgr.4108 v2:172.21.15.71:6824/2133533992 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ff9fc0ac570 con 0x7ff9f0036420 2024-04-19T01:48:58.392 INFO:teuthology.orchestra.run.smithi062.stderr:instructed osd(s) 3 to deep-scrub 2024-04-19T01:48:58.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.397+0000 7ffa14284640 1 -- 172.21.15.62:0/1581834028 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7ff9f0036420 msgr2=0x7ff9f00388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:58.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.397+0000 7ffa14284640 1 --2- 172.21.15.62:0/1581834028 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7ff9f0036420 0x7ff9f00388e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc0ab000 tx=0x7ff9fc008040 comp rx=0 tx=0).stop 2024-04-19T01:48:58.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.397+0000 7ffa14284640 10 monclient: shutdown 2024-04-19T01:48:58.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.397+0000 7ffa14284640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:58.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.397+0000 7ffa14284640 1 -- 172.21.15.62:0/1581834028 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa0c0a2ef0 msgr2=0x7ffa0c1cf970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:58.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.397+0000 7ffa14284640 1 --2- 172.21.15.62:0/1581834028 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa0c0a2ef0 0x7ffa0c1cf970 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ffa0800d5f0 tx=0x7ffa0800dab0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.397+0000 7ffa14284640 1 -- 172.21.15.62:0/1581834028 shutdown_connections 2024-04-19T01:48:58.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.397+0000 7ffa14284640 1 --2- 172.21.15.62:0/1581834028 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7ff9f0036420 0x7ff9f00388e0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.397+0000 7ffa14284640 1 --2- 172.21.15.62:0/1581834028 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7ffa0c0a4d20 0x7ffa0c1331f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.397+0000 7ffa14284640 1 --2- 172.21.15.62:0/1581834028 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa0c137ee0 0x7ffa0c13c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.397+0000 7ffa14284640 1 --2- 172.21.15.62:0/1581834028 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa0c0a2ef0 0x7ffa0c1cf970 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.397+0000 7ffa14284640 1 -- 172.21.15.62:0/1581834028 >> 172.21.15.62:0/1581834028 conn(0x7ffa0c0a23d0 msgr2=0x7ffa0c1362c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:58.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.397+0000 7ffa14284640 1 -- 172.21.15.62:0/1581834028 shutdown_connections 2024-04-19T01:48:58.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.397+0000 7ffa14284640 1 -- 172.21.15.62:0/1581834028 wait complete. 2024-04-19T01:48:58.411 INFO:tasks.ceph:Scrubbing osd.4 2024-04-19T01:48:58.411 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-19T01:48:58.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b36f48640 10 monclient: get_monmap_and_config 2024-04-19T01:48:58.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b36f48640 10 monclient: build_initial_monmap 2024-04-19T01:48:58.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b36f48640 10 monclient: monmap: 2024-04-19T01:48:58.532 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:58.533 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:58.533 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:48:58.538689+0000 2024-04-19T01:48:58.533 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:48:58.538689+0000 2024-04-19T01:48:58.533 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:58.533 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:58.533 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:48:58.533 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:48:58.533 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:48:58.533 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:48:58.533 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:48:58.533 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:58.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b36f48640 1 Processor -- start 2024-04-19T01:48:58.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b36f48640 1 -- start start 2024-04-19T01:48:58.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b36f48640 10 monclient: init 2024-04-19T01:48:58.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b36f48640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:58.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b36f48640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-04-19T01:48:58.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b36f48640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1b300a10a0 0x7f1b300a1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:58.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b36f48640 10 monclient(hunting): picked mon.noname-c con 0x7f1b300a10a0 addr v2:172.21.15.71:3300/0 2024-04-19T01:48:58.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b36f48640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1b300a19c0 0x7f1b30139820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:58.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b36f48640 10 monclient(hunting): picked mon.noname-a con 0x7f1b300a19c0 addr v2:172.21.15.62:3300/0 2024-04-19T01:48:58.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b36f48640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b30139d60 0x7f1b3013c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:58.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b36f48640 10 monclient(hunting): picked mon.noname-b con 0x7f1b30139d60 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:58.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b36f48640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:58.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b36f48640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f1b30099150 con 0x7f1b300a19c0 2024-04-19T01:48:58.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b36f48640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:58.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b36f48640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f1b3009dfc0 con 0x7f1b300a10a0 2024-04-19T01:48:58.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b36f48640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:58.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b36f48640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f1b3009e580 con 0x7f1b30139d60 2024-04-19T01:48:58.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b36f48640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:58.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b36f48640 10 monclient(hunting): authenticate will time out at 1324.358051s 2024-04-19T01:48:58.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b2ffff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1b300a19c0 0x7f1b30139820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:58.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b354e5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b30139d60 0x7f1b3013c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:58.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b354e5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b30139d60 0x7f1b3013c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:40028/0 (socket says 172.21.15.62:40028) 2024-04-19T01:48:58.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b2ffff640 10 monclient(hunting): get_auth_request con 0x7f1b300a19c0 auth_method 0 2024-04-19T01:48:58.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b354e5640 1 -- 172.21.15.62:0/2431302991 learned_addr learned my addr 172.21.15.62:0/2431302991 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:58.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:58.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b2ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:58.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:58.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b354e5640 10 monclient(hunting): get_auth_request con 0x7f1b30139d60 auth_method 0 2024-04-19T01:48:58.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b354e5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:58.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b354e5640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:58.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b354e5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:58.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b354e5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:58.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b354e5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:58.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b34ce4640 1 --2- 172.21.15.62:0/2431302991 >> v2:172.21.15.71:3300/0 conn(0x7f1b300a10a0 0x7f1b300a1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:58.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b354e5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:58.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:58.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:58.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:58.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b34ce4640 10 monclient(hunting): get_auth_request con 0x7f1b300a10a0 auth_method 0 2024-04-19T01:48:58.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b34ce4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:58.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b34ce4640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:58.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b34ce4640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:58.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b354e5640 10 monclient(hunting): handle_auth_done global_id 34178 payload 274 2024-04-19T01:48:58.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b354e5640 1 -- 172.21.15.62:0/2431302991 >> v2:172.21.15.71:3300/0 conn(0x7f1b300a10a0 msgr2=0x7f1b300a1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:58.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b354e5640 1 --2- 172.21.15.62:0/2431302991 >> v2:172.21.15.71:3300/0 conn(0x7f1b300a10a0 0x7f1b300a1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b354e5640 1 -- 172.21.15.62:0/2431302991 >> v2:172.21.15.62:3300/0 conn(0x7f1b300a19c0 msgr2=0x7f1b30139820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:58.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b354e5640 1 --2- 172.21.15.62:0/2431302991 >> v2:172.21.15.62:3300/0 conn(0x7f1b300a19c0 0x7f1b30139820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b354e5640 10 monclient: _finish_hunting 0 2024-04-19T01:48:58.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b354e5640 1 monclient: found mon.noname-b 2024-04-19T01:48:58.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b354e5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:58.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b354e5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:48:58.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b354e5640 1 -- 172.21.15.62:0/2431302991 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b3009f590 con 0x7f1b30139d60 2024-04-19T01:48:58.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.537+0000 7f1b2ffff640 1 --2- 172.21.15.62:0/2431302991 >> v2:172.21.15.62:3300/0 conn(0x7f1b300a19c0 0x7f1b30139820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:48:58.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.541+0000 7f1b354e5640 1 -- 172.21.15.62:0/2431302991 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b30139d60 msgr2=0x7f1b3013c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:58.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.541+0000 7f1b354e5640 1 --2- 172.21.15.62:0/2431302991 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b30139d60 0x7f1b3013c150 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1b3006ce30 tx=0x7f1b2801c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e36afab37b3a0dcc server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:58.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.541+0000 7f1b2f7fe640 1 -- 172.21.15.62:0/2431302991 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b280a5530 con 0x7f1b30139d60 2024-04-19T01:48:58.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.541+0000 7f1b2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:58.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.541+0000 7f1b2f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:48:58.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.541+0000 7f1b2f7fe640 10 monclient: dump: 2024-04-19T01:48:58.547 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:58.547 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:58.547 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:58.547 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:58.547 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:58.547 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:58.547 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:58.548 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:58.548 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:58.548 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:58.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.541+0000 7f1b2f7fe640 10 monclient: _finish_auth 0 2024-04-19T01:48:58.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.541+0000 7f1b2f7fe640 10 monclient: _check_auth_tickets 2024-04-19T01:48:58.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.541+0000 7f1b2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:58.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.541+0000 7f1b2f7fe640 1 -- 172.21.15.62:0/2431302991 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b280a56d0 con 0x7f1b30139d60 2024-04-19T01:48:58.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.541+0000 7f1b2f7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:58.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.541+0000 7f1b2f7fe640 1 -- 172.21.15.62:0/2431302991 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b280a5aa0 con 0x7f1b30139d60 2024-04-19T01:48:58.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.541+0000 7f1b36f48640 5 monclient: authenticate success, global_id 34178 2024-04-19T01:48:58.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.541+0000 7f1b36f48640 10 monclient: get_monmap_and_config success 2024-04-19T01:48:58.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.541+0000 7f1b36f48640 4 set_mon_vals no callback set 2024-04-19T01:48:58.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.541+0000 7f1b2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:58.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.541+0000 7f1b2f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:48:58.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.541+0000 7f1b2f7fe640 10 monclient: dump: 2024-04-19T01:48:58.549 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:58.549 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:58.549 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:58.549 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:58.549 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:58.549 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:58.549 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:58.549 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:58.549 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:58.549 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:58.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.541+0000 7f1b36f48640 10 monclient: shutdown 2024-04-19T01:48:58.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.541+0000 7f1b36f48640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:58.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.541+0000 7f1b36f48640 1 -- 172.21.15.62:0/2431302991 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b30139d60 msgr2=0x7f1b3013c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:58.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.541+0000 7f1b36f48640 1 --2- 172.21.15.62:0/2431302991 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b30139d60 0x7f1b3013c150 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1b3006ce30 tx=0x7f1b2801c8b0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.541+0000 7f1b36f48640 1 -- 172.21.15.62:0/2431302991 shutdown_connections 2024-04-19T01:48:58.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.541+0000 7f1b36f48640 1 --2- 172.21.15.62:0/2431302991 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b30139d60 0x7f1b3013c150 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.541+0000 7f1b36f48640 1 --2- 172.21.15.62:0/2431302991 >> v2:172.21.15.62:3300/0 conn(0x7f1b300a19c0 0x7f1b30139820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.541+0000 7f1b36f48640 1 --2- 172.21.15.62:0/2431302991 >> v2:172.21.15.71:3300/0 conn(0x7f1b300a10a0 0x7f1b300a1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.541+0000 7f1b36f48640 1 -- 172.21.15.62:0/2431302991 >> 172.21.15.62:0/2431302991 conn(0x7f1b300a2190 msgr2=0x7f1b300a25a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:58.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.541+0000 7f1b36f48640 1 -- 172.21.15.62:0/2431302991 shutdown_connections 2024-04-19T01:48:58.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.541+0000 7f1b36f48640 1 -- 172.21.15.62:0/2431302991 wait complete. 2024-04-19T01:48:58.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b36f48640 10 monclient: build_initial_monmap 2024-04-19T01:48:58.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b36f48640 10 monclient: monmap: 2024-04-19T01:48:58.550 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:58.550 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:48:58.550 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:48:58.550 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:48:58.550 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:58.550 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:58.551 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:48:58.551 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:48:58.551 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:48:58.551 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:58.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b36f48640 1 Processor -- start 2024-04-19T01:48:58.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b36f48640 1 -- start start 2024-04-19T01:48:58.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b36f48640 10 monclient: init 2024-04-19T01:48:58.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b36f48640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:58.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b36f48640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-19T01:48:58.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b36f48640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b300a10a0 0x7f1b301cd020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:58.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b36f48640 10 monclient(hunting): picked mon.noname-b con 0x7f1b300a10a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:48:58.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b36f48640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b300a19c0 0x7f1b30098220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:58.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b36f48640 10 monclient(hunting): picked mon.noname-c con 0x7f1b300a19c0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:58.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b36f48640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b30139d60 0x7f1b30098760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:58.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b36f48640 10 monclient(hunting): picked mon.noname-a con 0x7f1b30139d60 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:48:58.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b36f48640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:58.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b36f48640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3009f0f0 con 0x7f1b30139d60 2024-04-19T01:48:58.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b36f48640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:58.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b36f48640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3009ef70 con 0x7f1b300a10a0 2024-04-19T01:48:58.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b36f48640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:58.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b36f48640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f1b3009d6c0 con 0x7f1b300a19c0 2024-04-19T01:48:58.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b36f48640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:58.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b36f48640 10 monclient(hunting): authenticate will time out at 1324.365700s 2024-04-19T01:48:58.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b2ffff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b300a19c0 0x7f1b30098220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:58.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b354e5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b30139d60 0x7f1b30098760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:58.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b2ffff640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b300a19c0 0x7f1b30098220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:40030/0 (socket says 172.21.15.62:40030) 2024-04-19T01:48:58.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b354e5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b30139d60 0x7f1b30098760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47182/0 (socket says 172.21.15.62:47182) 2024-04-19T01:48:58.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b2ffff640 1 -- 172.21.15.62:0/2769885539 learned_addr learned my addr 172.21.15.62:0/2769885539 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:58.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b2ffff640 10 monclient(hunting): get_auth_request con 0x7f1b300a19c0 auth_method 0 2024-04-19T01:48:58.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:58.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b2ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:58.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:58.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b354e5640 10 monclient(hunting): get_auth_request con 0x7f1b30139d60 auth_method 0 2024-04-19T01:48:58.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b354e5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:58.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b354e5640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:58.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b354e5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:58.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b354e5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:58.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b354e5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:58.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b354e5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:58.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:58.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:58.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:58.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b34ce4640 1 --2- 172.21.15.62:0/2769885539 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b300a10a0 0x7f1b301cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:58.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b34ce4640 10 monclient(hunting): get_auth_request con 0x7f1b300a10a0 auth_method 0 2024-04-19T01:48:58.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b34ce4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:58.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b34ce4640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:58.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b34ce4640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:58.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b2ffff640 10 monclient(hunting): handle_auth_done global_id 34181 payload 824 2024-04-19T01:48:58.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b2ffff640 1 -- 172.21.15.62:0/2769885539 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b300a10a0 msgr2=0x7f1b301cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:58.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b2ffff640 1 --2- 172.21.15.62:0/2769885539 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b300a10a0 0x7f1b301cd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b2ffff640 1 -- 172.21.15.62:0/2769885539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b30139d60 msgr2=0x7f1b30098760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:58.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b2ffff640 1 --2- 172.21.15.62:0/2769885539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b30139d60 0x7f1b30098760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b2ffff640 10 monclient: _finish_hunting 0 2024-04-19T01:48:58.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b2ffff640 1 monclient: found mon.noname-c 2024-04-19T01:48:58.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:58.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b2ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:48:58.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b2ffff640 1 -- 172.21.15.62:0/2769885539 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b3009f380 con 0x7f1b300a19c0 2024-04-19T01:48:58.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.545+0000 7f1b354e5640 1 --2- 172.21.15.62:0/2769885539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b30139d60 0x7f1b30098760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:48:58.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b2ffff640 1 -- 172.21.15.62:0/2769885539 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b300a19c0 msgr2=0x7f1b30098220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:58.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b2ffff640 1 --2- 172.21.15.62:0/2769885539 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b300a19c0 0x7f1b30098220 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1b2400ebe0 tx=0x7f1b2400c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:58.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b2d7fa640 1 -- 172.21.15.62:0/2769885539 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b24017070 con 0x7f1b300a19c0 2024-04-19T01:48:58.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b2d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:58.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b2d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:48:58.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b2d7fa640 10 monclient: dump: 2024-04-19T01:48:58.555 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:58.555 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:58.555 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:58.555 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:58.555 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:58.555 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:58.555 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:58.555 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:58.555 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:58.555 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:58.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b2d7fa640 10 monclient: _finish_auth 0 2024-04-19T01:48:58.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b2d7fa640 10 monclient: _check_auth_tickets 2024-04-19T01:48:58.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b2d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:58.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b2d7fa640 1 -- 172.21.15.62:0/2769885539 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b2400ce40 con 0x7f1b300a19c0 2024-04-19T01:48:58.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b2d7fa640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:58.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b2d7fa640 1 -- 172.21.15.62:0/2769885539 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b24013690 con 0x7f1b300a19c0 2024-04-19T01:48:58.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b2d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:58.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b2d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:48:58.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b2d7fa640 10 monclient: dump: 2024-04-19T01:48:58.556 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:58.556 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:58.556 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:58.556 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:58.556 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:58.556 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:58.556 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:58.557 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:58.557 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:58.557 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:58.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b2e7fc640 4 set_mon_vals no callback set 2024-04-19T01:48:58.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b36f48640 5 monclient: authenticate success, global_id 34181 2024-04-19T01:48:58.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b36f48640 10 monclient: _renew_subs 2024-04-19T01:48:58.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b36f48640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:48:58.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b36f48640 1 -- 172.21.15.62:0/2769885539 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b3013a4e0 con 0x7f1b300a19c0 2024-04-19T01:48:58.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b36f48640 10 monclient: _renew_subs 2024-04-19T01:48:58.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b36f48640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:48:58.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b36f48640 1 -- 172.21.15.62:0/2769885539 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b300906f0 con 0x7f1b300a19c0 2024-04-19T01:48:58.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b36f48640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:48:58.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b36f48640 1 -- 172.21.15.62:0/2769885539 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1b3004fcf0 con 0x7f1b300a19c0 2024-04-19T01:48:58.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b2d7fa640 1 -- 172.21.15.62:0/2769885539 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f1b2401a070 con 0x7f1b300a19c0 2024-04-19T01:48:58.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b2d7fa640 1 --2- 172.21.15.62:0/2769885539 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f1b1c036450 0x7f1b1c038910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:58.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b2d7fa640 1 -- 172.21.15.62:0/2769885539 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(325..325 src has 115..325) ==== 4346+0+0 (secure 0 0 0) 0x7f1b240492f0 con 0x7f1b300a19c0 2024-04-19T01:48:58.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b2d7fa640 1 --2- 172.21.15.62:0/2769885539 >> [v2:172.21.15.71:6808/3342848064,v1:172.21.15.71:6809/3342848064] conn(0x7f1b1c03c250 0x7f1b1c03e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:58.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b2d7fa640 1 -- 172.21.15.62:0/2769885539 --> [v2:172.21.15.71:6808/3342848064,v1:172.21.15.71:6809/3342848064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1b2400be50 con 0x7f1b1c03c250 2024-04-19T01:48:58.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b2d7fa640 1 -- 172.21.15.62:0/2769885539 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_get_version_reply(handle=1 version=325) ==== 24+0+0 (secure 0 0 0) 0x7f1b24050050 con 0x7f1b300a19c0 2024-04-19T01:48:58.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b2d7fa640 10 monclient: handle_get_version_reply finishing 1 version 325 2024-04-19T01:48:58.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b34ce4640 1 --2- 172.21.15.62:0/2769885539 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f1b1c036450 0x7f1b1c038910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:58.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b34ce4640 10 monclient: get_auth_request con 0x7f1b1c036450 auth_method 0 2024-04-19T01:48:58.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b354e5640 1 --2- 172.21.15.62:0/2769885539 >> [v2:172.21.15.71:6808/3342848064,v1:172.21.15.71:6809/3342848064] conn(0x7f1b1c03c250 0x7f1b1c03e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:58.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.549+0000 7f1b354e5640 10 monclient: get_auth_request con 0x7f1b1c03c250 auth_method 0 2024-04-19T01:48:58.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.553+0000 7f1b34ce4640 1 --2- 172.21.15.62:0/2769885539 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f1b1c036450 0x7f1b1c038910 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f1b2000b9d0 tx=0x7f1b20008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:58.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.553+0000 7f1b354e5640 1 --2- 172.21.15.62:0/2769885539 >> [v2:172.21.15.71:6808/3342848064,v1:172.21.15.71:6809/3342848064] conn(0x7f1b1c03c250 0x7f1b1c03e670 crc :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:58.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.557+0000 7f1b2d7fa640 1 -- 172.21.15.62:0/2769885539 <== osd.4 v2:172.21.15.71:6808/3342848064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1b2400be50 con 0x7f1b1c03c250 2024-04-19T01:48:58.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.569+0000 7f1b36f48640 1 -- 172.21.15.62:0/2769885539 --> [v2:172.21.15.71:6808/3342848064,v1:172.21.15.71:6809/3342848064] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f1b3008f030 con 0x7f1b1c03c250 2024-04-19T01:48:58.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.569+0000 7f1b2d7fa640 1 -- 172.21.15.62:0/2769885539 <== osd.4 v2:172.21.15.71:6808/3342848064 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f1b3008f030 con 0x7f1b1c03c250 2024-04-19T01:48:58.567 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-19T01:48:58.567 INFO:teuthology.orchestra.run.smithi062.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T01:48:58.568 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-19T01:48:58.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.573+0000 7f1b36f48640 1 -- 172.21.15.62:0/2769885539 >> [v2:172.21.15.71:6808/3342848064,v1:172.21.15.71:6809/3342848064] conn(0x7f1b1c03c250 msgr2=0x7f1b1c03e670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:58.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.573+0000 7f1b36f48640 1 --2- 172.21.15.62:0/2769885539 >> [v2:172.21.15.71:6808/3342848064,v1:172.21.15.71:6809/3342848064] conn(0x7f1b1c03c250 0x7f1b1c03e670 crc :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.573+0000 7f1b36f48640 1 -- 172.21.15.62:0/2769885539 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f1b1c036450 msgr2=0x7f1b1c038910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:58.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.573+0000 7f1b36f48640 1 --2- 172.21.15.62:0/2769885539 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f1b1c036450 0x7f1b1c038910 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f1b2000b9d0 tx=0x7f1b20008040 comp rx=0 tx=0).stop 2024-04-19T01:48:58.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.573+0000 7f1b36f48640 10 monclient: shutdown 2024-04-19T01:48:58.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.573+0000 7f1b36f48640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:58.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.573+0000 7f1b36f48640 1 -- 172.21.15.62:0/2769885539 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b300a19c0 msgr2=0x7f1b30098220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:58.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.573+0000 7f1b36f48640 1 --2- 172.21.15.62:0/2769885539 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f1b300a19c0 0x7f1b30098220 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1b2400ebe0 tx=0x7f1b2400c520 comp rx=0 tx=0).stop 2024-04-19T01:48:58.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.573+0000 7f1b354e5640 1 -- 172.21.15.62:0/2769885539 reap_dead start 2024-04-19T01:48:58.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.573+0000 7f1b36f48640 1 -- 172.21.15.62:0/2769885539 shutdown_connections 2024-04-19T01:48:58.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.573+0000 7f1b36f48640 1 -- 172.21.15.62:0/2769885539 >> 172.21.15.62:0/2769885539 conn(0x7f1b300a2190 msgr2=0x7f1b300a4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:58.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.573+0000 7f1b36f48640 1 -- 172.21.15.62:0/2769885539 shutdown_connections 2024-04-19T01:48:58.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.573+0000 7f1b36f48640 1 -- 172.21.15.62:0/2769885539 wait complete. 2024-04-19T01:48:58.583 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-19T01:48:58.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f88332c1640 10 monclient: get_monmap_and_config 2024-04-19T01:48:58.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f88332c1640 10 monclient: build_initial_monmap 2024-04-19T01:48:58.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f88332c1640 10 monclient: monmap: 2024-04-19T01:48:58.692 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:58.692 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:58.692 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:48:58.698339+0000 2024-04-19T01:48:58.692 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:48:58.698339+0000 2024-04-19T01:48:58.692 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:58.692 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:58.693 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:48:58.693 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:48:58.693 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:48:58.693 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:48:58.693 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:48:58.693 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:58.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f88332c1640 1 Processor -- start 2024-04-19T01:48:58.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f88332c1640 1 -- start start 2024-04-19T01:48:58.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f88332c1640 10 monclient: init 2024-04-19T01:48:58.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f88332c1640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:58.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f88332c1640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-04-19T01:48:58.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f88332c1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f882c137780 addr v1:172.21.15.62:6789/0 2024-04-19T01:48:58.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f88332c1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f882c0a60b0 0x7f882c13c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:58.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f88332c1640 10 monclient(hunting): picked mon.noname-b con 0x7f882c0a60b0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:58.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f88332c1640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f882c13cc60 0x7f882c13f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:58.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f88332c1640 10 monclient(hunting): picked mon.noname-c con 0x7f882c13cc60 addr v2:172.21.15.71:3300/0 2024-04-19T01:48:58.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f88332c1640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f882c09c2e0 con 0x7f882c137780 2024-04-19T01:48:58.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f88332c1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:58.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f88332c1640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f882c09bd20 con 0x7f882c13cc60 2024-04-19T01:48:58.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f88332c1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:58.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f88332c1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f882c09c000 con 0x7f882c0a60b0 2024-04-19T01:48:58.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f88332c1640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:58.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f88332c1640 10 monclient(hunting): authenticate will time out at 1324.517854s 2024-04-19T01:48:58.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f883105d640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f882c137780 0x7f882c137b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53268/0 (socket says 172.21.15.62:53268) 2024-04-19T01:48:58.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f883085c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f882c0a60b0 0x7f882c13c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:58.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f883105d640 1 -- 172.21.15.62:0/430626733 learned_addr learned my addr 172.21.15.62:0/430626733 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:48:58.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f883085c640 10 monclient(hunting): get_auth_request con 0x7f882c0a60b0 auth_method 0 2024-04-19T01:48:58.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f883085c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:58.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f883085c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:58.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f883085c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:58.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f883085c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:58.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f883085c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:58.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f883185e640 1 --2- 172.21.15.62:0/430626733 >> v2:172.21.15.71:3300/0 conn(0x7f882c13cc60 0x7f882c13f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:58.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f883085c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:58.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f883185e640 10 monclient(hunting): get_auth_request con 0x7f882c13cc60 auth_method 0 2024-04-19T01:48:58.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f883185e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:58.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f883185e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:58.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f883185e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:58.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f883185e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:58.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f883185e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:58.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f883185e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:58.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f883085c640 10 monclient(hunting): handle_auth_done global_id 34184 payload 274 2024-04-19T01:48:58.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f883085c640 1 -- 172.21.15.62:0/430626733 >> v2:172.21.15.71:3300/0 conn(0x7f882c13cc60 msgr2=0x7f882c13f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:58.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f883085c640 1 --2- 172.21.15.62:0/430626733 >> v2:172.21.15.71:3300/0 conn(0x7f882c13cc60 0x7f882c13f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f883085c640 1 -- 172.21.15.62:0/430626733 >> v1:172.21.15.62:6789/0 conn(0x7f882c137780 legacy=0x7f882c137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:58.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f883085c640 10 monclient: _finish_hunting 0 2024-04-19T01:48:58.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f883085c640 1 monclient: found mon.noname-b 2024-04-19T01:48:58.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f883085c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:58.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f883085c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:48:58.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.697+0000 7f883085c640 1 -- 172.21.15.62:0/430626733 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f882c138270 con 0x7f882c0a60b0 2024-04-19T01:48:58.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f883085c640 1 -- 172.21.15.62:0/430626733 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f882c0a60b0 msgr2=0x7f882c13c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:58.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f883085c640 1 --2- 172.21.15.62:0/430626733 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f882c0a60b0 0x7f882c13c720 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f88200097f0 tx=0x7f882001c180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60aaee57e9615731 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:58.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f882bfff640 1 -- 172.21.15.62:0/430626733 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88200b0070 con 0x7f882c0a60b0 2024-04-19T01:48:58.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f882bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:58.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f882bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:48:58.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f882bfff640 10 monclient: dump: 2024-04-19T01:48:58.702 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:58.702 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:58.702 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:58.702 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:58.702 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:58.702 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:58.702 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:58.702 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:58.702 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:58.702 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:58.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f882bfff640 10 monclient: _finish_auth 0 2024-04-19T01:48:58.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f882bfff640 10 monclient: _check_auth_tickets 2024-04-19T01:48:58.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f882bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:58.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 5 monclient: authenticate success, global_id 34184 2024-04-19T01:48:58.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f882bfff640 1 -- 172.21.15.62:0/430626733 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88200ab600 con 0x7f882c0a60b0 2024-04-19T01:48:58.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:48:58.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f882bfff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:58.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 10 monclient: get_monmap_and_config success 2024-04-19T01:48:58.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 4 set_mon_vals no callback set 2024-04-19T01:48:58.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f882bfff640 1 -- 172.21.15.62:0/430626733 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88200ab9d0 con 0x7f882c0a60b0 2024-04-19T01:48:58.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f882bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:58.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f882bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:48:58.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f882bfff640 10 monclient: dump: 2024-04-19T01:48:58.703 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:58.703 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:58.703 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:58.703 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:58.703 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:58.703 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:58.703 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:58.704 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:58.704 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:58.704 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:58.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 10 monclient: shutdown 2024-04-19T01:48:58.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:58.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 1 -- 172.21.15.62:0/430626733 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f882c0a60b0 msgr2=0x7f882c13c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:58.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 1 --2- 172.21.15.62:0/430626733 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f882c0a60b0 0x7f882c13c720 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f88200097f0 tx=0x7f882001c180 comp rx=0 tx=0).stop 2024-04-19T01:48:58.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 1 -- 172.21.15.62:0/430626733 shutdown_connections 2024-04-19T01:48:58.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 1 --2- 172.21.15.62:0/430626733 >> v2:172.21.15.71:3300/0 conn(0x7f882c13cc60 0x7f882c13f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 1 --2- 172.21.15.62:0/430626733 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f882c0a60b0 0x7f882c13c720 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 1 -- 172.21.15.62:0/430626733 >> 172.21.15.62:0/430626733 conn(0x7f882c0aa1d0 msgr2=0x7f882c0a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:58.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 1 -- 172.21.15.62:0/430626733 shutdown_connections 2024-04-19T01:48:58.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 1 -- 172.21.15.62:0/430626733 wait complete. 2024-04-19T01:48:58.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 10 monclient: build_initial_monmap 2024-04-19T01:48:58.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 10 monclient: monmap: 2024-04-19T01:48:58.705 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:58.705 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:48:58.705 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:48:58.705 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:48:58.705 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:58.705 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:58.705 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:48:58.705 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:48:58.705 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:48:58.705 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:58.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 1 Processor -- start 2024-04-19T01:48:58.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 1 -- start start 2024-04-19T01:48:58.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 10 monclient: init 2024-04-19T01:48:58.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:58.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-19T01:48:58.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f882c0a60b0 0x7f882c1cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:58.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 10 monclient(hunting): picked mon.noname-a con 0x7f882c0a60b0 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:48:58.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f882c137780 0x7f882c1364a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:58.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 10 monclient(hunting): picked mon.noname-b con 0x7f882c137780 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:48:58.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f882c13cc60 0x7f882c09f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:58.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 10 monclient(hunting): picked mon.noname-c con 0x7f882c13cc60 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:58.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:58.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f882c142140 con 0x7f882c0a60b0 2024-04-19T01:48:58.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:58.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f882c141fc0 con 0x7f882c137780 2024-04-19T01:48:58.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:58.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f882c1422c0 con 0x7f882c13cc60 2024-04-19T01:48:58.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:58.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f88332c1640 10 monclient(hunting): authenticate will time out at 1324.526502s 2024-04-19T01:48:58.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f883105d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f882c0a60b0 0x7f882c1cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:58.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f883105d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f882c0a60b0 0x7f882c1cf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47190/0 (socket says 172.21.15.62:47190) 2024-04-19T01:48:58.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f883105d640 1 -- 172.21.15.62:0/2646562326 learned_addr learned my addr 172.21.15.62:0/2646562326 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:58.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f883105d640 10 monclient(hunting): get_auth_request con 0x7f882c0a60b0 auth_method 0 2024-04-19T01:48:58.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f883105d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:58.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f883105d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:58.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f883105d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:58.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f883185e640 1 --2- 172.21.15.62:0/2646562326 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f882c13cc60 0x7f882c09f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:58.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f883105d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:58.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f883105d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:58.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f883105d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:58.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f883185e640 10 monclient(hunting): get_auth_request con 0x7f882c13cc60 auth_method 0 2024-04-19T01:48:58.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f883185e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:58.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f883185e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:58.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f883185e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:58.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f883185e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:58.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f883185e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:58.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f883185e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:58.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f883105d640 10 monclient(hunting): handle_auth_done global_id 15357 payload 824 2024-04-19T01:48:58.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f883105d640 1 -- 172.21.15.62:0/2646562326 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f882c13cc60 msgr2=0x7f882c09f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:58.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f883105d640 1 --2- 172.21.15.62:0/2646562326 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f882c13cc60 0x7f882c09f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f883085c640 1 --2- 172.21.15.62:0/2646562326 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f882c137780 0x7f882c1364a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:58.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f883105d640 1 -- 172.21.15.62:0/2646562326 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f882c137780 msgr2=0x7f882c1364a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:58.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f883105d640 1 --2- 172.21.15.62:0/2646562326 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f882c137780 0x7f882c1364a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f883105d640 10 monclient: _finish_hunting 0 2024-04-19T01:48:58.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f883105d640 1 monclient: found mon.noname-a 2024-04-19T01:48:58.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f883105d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:58.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f883105d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:48:58.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.705+0000 7f883105d640 1 -- 172.21.15.62:0/2646562326 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f882c120b50 con 0x7f882c0a60b0 2024-04-19T01:48:58.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.709+0000 7f883105d640 1 -- 172.21.15.62:0/2646562326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f882c0a60b0 msgr2=0x7f882c1cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:58.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.709+0000 7f883105d640 1 --2- 172.21.15.62:0/2646562326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f882c0a60b0 0x7f882c1cf4d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f882400cc10 tx=0x7f882400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:58.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.709+0000 7f8829ffb640 1 -- 172.21.15.62:0/2646562326 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8824012400 con 0x7f882c0a60b0 2024-04-19T01:48:58.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.709+0000 7f8829ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:58.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.709+0000 7f8829ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:48:58.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.709+0000 7f8829ffb640 10 monclient: dump: 2024-04-19T01:48:58.713 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:58.713 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:58.713 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:58.713 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:58.713 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:58.713 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:58.713 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:58.713 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:58.713 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:58.713 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:58.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.709+0000 7f8829ffb640 10 monclient: _finish_auth 0 2024-04-19T01:48:58.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.709+0000 7f8829ffb640 10 monclient: _check_auth_tickets 2024-04-19T01:48:58.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.709+0000 7f8829ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:58.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.709+0000 7f8829ffb640 1 -- 172.21.15.62:0/2646562326 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88240125e0 con 0x7f882c0a60b0 2024-04-19T01:48:58.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.709+0000 7f88332c1640 5 monclient: authenticate success, global_id 15357 2024-04-19T01:48:58.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.709+0000 7f88332c1640 10 monclient: _renew_subs 2024-04-19T01:48:58.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.709+0000 7f88332c1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:58.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.709+0000 7f88332c1640 1 -- 172.21.15.62:0/2646562326 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f882c120e30 con 0x7f882c0a60b0 2024-04-19T01:48:58.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.709+0000 7f8829ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:58.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.709+0000 7f8829ffb640 1 -- 172.21.15.62:0/2646562326 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88240129d0 con 0x7f882c0a60b0 2024-04-19T01:48:58.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.709+0000 7f8829ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:58.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.709+0000 7f8829ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:48:58.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.709+0000 7f8829ffb640 10 monclient: dump: 2024-04-19T01:48:58.714 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:58.714 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:58.714 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:58.714 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:58.714 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:58.714 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:58.715 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.709+0000 7f88332c1640 10 monclient: _renew_subs 2024-04-19T01:48:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.709+0000 7f88332c1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.709+0000 7f88332c1640 1 -- 172.21.15.62:0/2646562326 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f882c0a1f90 con 0x7f882c0a60b0 2024-04-19T01:48:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.709+0000 7f882b7fe640 4 set_mon_vals no callback set 2024-04-19T01:48:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.709+0000 7f88332c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.709+0000 7f88332c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.709+0000 7f88332c1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.709+0000 7f88332c1640 1 -- 172.21.15.62:0/2646562326 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f882c068ab0 con 0x7f882c0a60b0 2024-04-19T01:48:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.709+0000 7f8829ffb640 1 -- 172.21.15.62:0/2646562326 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f8824012bf0 con 0x7f882c0a60b0 2024-04-19T01:48:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.709+0000 7f8829ffb640 1 --2- 172.21.15.62:0/2646562326 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f8818036420 0x7f88180388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.709+0000 7f8829ffb640 1 -- 172.21.15.62:0/2646562326 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(325..325 src has 115..325) ==== 4346+0+0 (secure 0 0 0) 0x7f8824049ab0 con 0x7f882c0a60b0 2024-04-19T01:48:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.709+0000 7f883085c640 1 --2- 172.21.15.62:0/2646562326 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f8818036420 0x7f88180388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:58.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.709+0000 7f883085c640 10 monclient: get_auth_request con 0x7f8818036420 auth_method 0 2024-04-19T01:48:58.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.713+0000 7f883085c640 1 --2- 172.21.15.62:0/2646562326 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f8818036420 0x7f88180388e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f882c0a5a70 tx=0x7f8820008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:58.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.713+0000 7f8829ffb640 1 -- 172.21.15.62:0/2646562326 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f882401a360 con 0x7f882c0a60b0 2024-04-19T01:48:58.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.713+0000 7f8829ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:58.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.713+0000 7f8829ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:48:58.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.881+0000 7f88332c1640 1 -- 172.21.15.62:0/2646562326 --> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f882c0a1430 con 0x7f8818036420 2024-04-19T01:48:58.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.881+0000 7f8829ffb640 1 -- 172.21.15.62:0/2646562326 <== mgr.4108 v2:172.21.15.71:6824/2133533992 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f882c0a1430 con 0x7f8818036420 2024-04-19T01:48:58.878 INFO:teuthology.orchestra.run.smithi062.stderr:instructed osd(s) 4 to deep-scrub 2024-04-19T01:48:58.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.885+0000 7f88332c1640 1 -- 172.21.15.62:0/2646562326 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f8818036420 msgr2=0x7f88180388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:58.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.885+0000 7f88332c1640 1 --2- 172.21.15.62:0/2646562326 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f8818036420 0x7f88180388e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f882c0a5a70 tx=0x7f8820008040 comp rx=0 tx=0).stop 2024-04-19T01:48:58.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.885+0000 7f88332c1640 10 monclient: shutdown 2024-04-19T01:48:58.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.885+0000 7f88332c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:58.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.885+0000 7f88332c1640 1 -- 172.21.15.62:0/2646562326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f882c0a60b0 msgr2=0x7f882c1cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:58.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.885+0000 7f88332c1640 1 --2- 172.21.15.62:0/2646562326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f882c0a60b0 0x7f882c1cf4d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f882400cc10 tx=0x7f882400a430 comp rx=0 tx=0).stop 2024-04-19T01:48:58.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.885+0000 7f88332c1640 1 -- 172.21.15.62:0/2646562326 shutdown_connections 2024-04-19T01:48:58.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.885+0000 7f88332c1640 1 --2- 172.21.15.62:0/2646562326 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f8818036420 0x7f88180388e0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.885+0000 7f88332c1640 1 --2- 172.21.15.62:0/2646562326 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f882c13cc60 0x7f882c09f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.885+0000 7f88332c1640 1 --2- 172.21.15.62:0/2646562326 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f882c137780 0x7f882c1364a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.885+0000 7f88332c1640 1 --2- 172.21.15.62:0/2646562326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f882c0a60b0 0x7f882c1cf4d0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:58.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.885+0000 7f88332c1640 1 -- 172.21.15.62:0/2646562326 >> 172.21.15.62:0/2646562326 conn(0x7f882c0aa1d0 msgr2=0x7f882c13d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:58.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.885+0000 7f88332c1640 1 -- 172.21.15.62:0/2646562326 shutdown_connections 2024-04-19T01:48:58.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:58.885+0000 7f88332c1640 1 -- 172.21.15.62:0/2646562326 wait complete. 2024-04-19T01:48:58.895 INFO:tasks.ceph:Scrubbing osd.5 2024-04-19T01:48:58.895 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-19T01:48:59.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.009+0000 7fb75bbb5640 10 monclient: get_monmap_and_config 2024-04-19T01:48:59.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.009+0000 7fb75bbb5640 10 monclient: build_initial_monmap 2024-04-19T01:48:59.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.009+0000 7fb75bbb5640 10 monclient: monmap: 2024-04-19T01:48:59.006 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:59.006 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:59.006 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:48:59.012279+0000 2024-04-19T01:48:59.006 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:48:59.012279+0000 2024-04-19T01:48:59.006 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:59.006 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:59.006 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:48:59.007 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:48:59.007 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:48:59.007 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:48:59.007 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:48:59.007 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:59.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.009+0000 7fb75bbb5640 1 Processor -- start 2024-04-19T01:48:59.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.009+0000 7fb75bbb5640 1 -- start start 2024-04-19T01:48:59.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.009+0000 7fb75bbb5640 10 monclient: init 2024-04-19T01:48:59.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb75bbb5640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:59.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb75bbb5640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-04-19T01:48:59.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb75bbb5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7540a10a0 0x7fb7540a1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:59.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb75bbb5640 10 monclient(hunting): picked mon.noname-b con 0x7fb7540a10a0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:59.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb75bbb5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb7540a19c0 addr v1:172.21.15.71:6789/0 2024-04-19T01:48:59.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb75bbb5640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb75413af40 0x7fb75413d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:59.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb75bbb5640 10 monclient(hunting): picked mon.noname-c con 0x7fb75413af40 addr v2:172.21.15.71:3300/0 2024-04-19T01:48:59.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb75bbb5640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb75409e580 con 0x7fb7540a19c0 2024-04-19T01:48:59.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb75bbb5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:59.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb75bbb5640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7fb754099150 con 0x7fb75413af40 2024-04-19T01:48:59.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb75bbb5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:59.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb75bbb5640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fb75409dfc0 con 0x7fb7540a10a0 2024-04-19T01:48:59.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb75bbb5640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:59.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb75bbb5640 10 monclient(hunting): authenticate will time out at 1324.832181s 2024-04-19T01:48:59.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb759951640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7540a10a0 0x7fb7540a1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:59.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb759951640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7540a10a0 0x7fb7540a1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:40046/0 (socket says 172.21.15.62:40046) 2024-04-19T01:48:59.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb759951640 1 -- 172.21.15.62:0/216275851 learned_addr learned my addr 172.21.15.62:0/216275851 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:59.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb759951640 10 monclient(hunting): get_auth_request con 0x7fb7540a10a0 auth_method 0 2024-04-19T01:48:59.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb759951640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:59.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb759951640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:59.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb759951640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:59.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb759951640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:59.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb759951640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:59.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb759951640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:59.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb75a152640 1 --2- 172.21.15.62:0/216275851 >> v2:172.21.15.71:3300/0 conn(0x7fb75413af40 0x7fb75413d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:59.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb759951640 10 monclient(hunting): handle_auth_done global_id 34190 payload 274 2024-04-19T01:48:59.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb759951640 1 -- 172.21.15.62:0/216275851 >> v2:172.21.15.71:3300/0 conn(0x7fb75413af40 msgr2=0x7fb75413d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:59.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb759951640 1 --2- 172.21.15.62:0/216275851 >> v2:172.21.15.71:3300/0 conn(0x7fb75413af40 0x7fb75413d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:59.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb759951640 1 -- 172.21.15.62:0/216275851 >> v1:172.21.15.71:6789/0 conn(0x7fb7540a19c0 legacy=0x7fb754139820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:59.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb759951640 10 monclient: _finish_hunting 0 2024-04-19T01:48:59.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb759951640 1 monclient: found mon.noname-b 2024-04-19T01:48:59.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb759951640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:59.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb759951640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:48:59.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb759951640 1 -- 172.21.15.62:0/216275851 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb75409f590 con 0x7fb7540a10a0 2024-04-19T01:48:59.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb75a152640 10 monclient: get_auth_request con 0x7fb75413af40 auth_method 0 2024-04-19T01:48:59.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.013+0000 7fb75a152640 1 --2- 172.21.15.62:0/216275851 >> v2:172.21.15.71:3300/0 conn(0x7fb75413af40 0x7fb75413d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:48:59.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb759951640 1 -- 172.21.15.62:0/216275851 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7540a10a0 msgr2=0x7fb7540a1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:59.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb759951640 1 --2- 172.21.15.62:0/216275851 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7540a10a0 0x7fb7540a1480 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb750009e50 tx=0x7fb75001c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8c640cf5c7cbbcc server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:59.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75894f640 1 -- 172.21.15.62:0/216275851 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7500a55c0 con 0x7fb7540a10a0 2024-04-19T01:48:59.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75894f640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:59.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75894f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:48:59.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75894f640 10 monclient: dump: 2024-04-19T01:48:59.014 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:59.014 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:59.014 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:59.014 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:59.014 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:59.014 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:59.014 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:59.014 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:59.014 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:59.014 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:59.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75894f640 10 monclient: _finish_auth 0 2024-04-19T01:48:59.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75894f640 10 monclient: _check_auth_tickets 2024-04-19T01:48:59.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75894f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:59.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75894f640 1 -- 172.21.15.62:0/216275851 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7500a5760 con 0x7fb7540a10a0 2024-04-19T01:48:59.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 5 monclient: authenticate success, global_id 34190 2024-04-19T01:48:59.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:48:59.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75894f640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:59.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 10 monclient: get_monmap_and_config success 2024-04-19T01:48:59.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 4 set_mon_vals no callback set 2024-04-19T01:48:59.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75894f640 1 -- 172.21.15.62:0/216275851 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7500a5b30 con 0x7fb7540a10a0 2024-04-19T01:48:59.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75894f640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:59.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75894f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:48:59.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75894f640 10 monclient: dump: 2024-04-19T01:48:59.015 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:59.015 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:59.016 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:59.016 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:59.016 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:59.016 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:59.016 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:59.016 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:59.016 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:59.016 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:59.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 10 monclient: shutdown 2024-04-19T01:48:59.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:59.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 1 -- 172.21.15.62:0/216275851 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7540a10a0 msgr2=0x7fb7540a1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:59.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 1 --2- 172.21.15.62:0/216275851 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7540a10a0 0x7fb7540a1480 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb750009e50 tx=0x7fb75001c8f0 comp rx=0 tx=0).stop 2024-04-19T01:48:59.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 1 -- 172.21.15.62:0/216275851 shutdown_connections 2024-04-19T01:48:59.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 1 --2- 172.21.15.62:0/216275851 >> v2:172.21.15.71:3300/0 conn(0x7fb75413af40 0x7fb75413d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:59.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 1 --2- 172.21.15.62:0/216275851 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7540a10a0 0x7fb7540a1480 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:59.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 1 -- 172.21.15.62:0/216275851 >> 172.21.15.62:0/216275851 conn(0x7fb7540a2190 msgr2=0x7fb7540a25a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:59.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 1 -- 172.21.15.62:0/216275851 shutdown_connections 2024-04-19T01:48:59.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 1 -- 172.21.15.62:0/216275851 wait complete. 2024-04-19T01:48:59.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 10 monclient: build_initial_monmap 2024-04-19T01:48:59.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 10 monclient: monmap: 2024-04-19T01:48:59.017 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:59.017 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:48:59.017 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:48:59.017 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:48:59.017 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:59.017 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:59.017 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:48:59.017 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:48:59.017 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:48:59.017 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:59.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 1 Processor -- start 2024-04-19T01:48:59.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 1 -- start start 2024-04-19T01:48:59.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 10 monclient: init 2024-04-19T01:48:59.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:59.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-19T01:48:59.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7540a10a0 0x7fb7541340c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:59.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 10 monclient(hunting): picked mon.noname-b con 0x7fb7540a10a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:48:59.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7540a19c0 0x7fb754099690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:59.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 10 monclient(hunting): picked mon.noname-c con 0x7fb7540a19c0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:59.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb75413af40 0x7fb754099bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:59.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 10 monclient(hunting): picked mon.noname-a con 0x7fb75413af40 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:48:59.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:59.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb75409f0f0 con 0x7fb75413af40 2024-04-19T01:48:59.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:59.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb75409ef70 con 0x7fb7540a10a0 2024-04-19T01:48:59.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:59.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7fb75409d6c0 con 0x7fb7540a19c0 2024-04-19T01:48:59.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:59.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75bbb5640 10 monclient(hunting): authenticate will time out at 1324.838798s 2024-04-19T01:48:59.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb759150640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7540a19c0 0x7fb754099690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:59.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb759150640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7540a19c0 0x7fb754099690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:40048/0 (socket says 172.21.15.62:40048) 2024-04-19T01:48:59.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75a152640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb75413af40 0x7fb754099bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:59.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb759150640 1 -- 172.21.15.62:0/3268055175 learned_addr learned my addr 172.21.15.62:0/3268055175 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:59.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb759150640 10 monclient(hunting): get_auth_request con 0x7fb7540a19c0 auth_method 0 2024-04-19T01:48:59.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb759150640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:59.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb759150640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:59.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb759150640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:59.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75a152640 10 monclient(hunting): get_auth_request con 0x7fb75413af40 auth_method 0 2024-04-19T01:48:59.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75a152640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:59.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75a152640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:59.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75a152640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:59.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb759150640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:59.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb759150640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:59.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb759150640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:59.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb759951640 1 --2- 172.21.15.62:0/3268055175 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7540a10a0 0x7fb7541340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:59.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb759951640 10 monclient(hunting): get_auth_request con 0x7fb7540a10a0 auth_method 0 2024-04-19T01:48:59.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb759951640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:59.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb759951640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:59.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb759951640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:59.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb759150640 10 monclient(hunting): handle_auth_done global_id 34193 payload 824 2024-04-19T01:48:59.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb759150640 1 -- 172.21.15.62:0/3268055175 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7540a10a0 msgr2=0x7fb7541340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:59.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb759150640 1 --2- 172.21.15.62:0/3268055175 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7540a10a0 0x7fb7541340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:59.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb759150640 1 -- 172.21.15.62:0/3268055175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb75413af40 msgr2=0x7fb754099bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:59.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb759150640 1 --2- 172.21.15.62:0/3268055175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb75413af40 0x7fb754099bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:59.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb759150640 10 monclient: _finish_hunting 0 2024-04-19T01:48:59.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb759150640 1 monclient: found mon.noname-c 2024-04-19T01:48:59.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb759150640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:59.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb759150640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:48:59.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb759150640 1 -- 172.21.15.62:0/3268055175 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb754071060 con 0x7fb7540a19c0 2024-04-19T01:48:59.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.017+0000 7fb75a152640 1 --2- 172.21.15.62:0/3268055175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb75413af40 0x7fb754099bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T01:48:59.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb759150640 1 -- 172.21.15.62:0/3268055175 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7540a19c0 msgr2=0x7fb754099690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:59.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb759150640 1 --2- 172.21.15.62:0/3268055175 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7540a19c0 0x7fb754099690 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb74400c450 tx=0x7fb74400c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:59.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb74a7fc640 1 -- 172.21.15.62:0/3268055175 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb744012a40 con 0x7fb7540a19c0 2024-04-19T01:48:59.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb74a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:59.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb74a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:48:59.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb74a7fc640 10 monclient: dump: 2024-04-19T01:48:59.025 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:59.025 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:59.025 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:59.025 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:59.025 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:59.025 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:59.025 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:59.025 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:59.026 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:59.026 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:59.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb74a7fc640 10 monclient: _finish_auth 0 2024-04-19T01:48:59.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb74a7fc640 10 monclient: _check_auth_tickets 2024-04-19T01:48:59.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb74a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:59.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb74a7fc640 1 -- 172.21.15.62:0/3268055175 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb744017070 con 0x7fb7540a19c0 2024-04-19T01:48:59.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb74a7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:59.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb75bbb5640 5 monclient: authenticate success, global_id 34193 2024-04-19T01:48:59.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb74a7fc640 1 -- 172.21.15.62:0/3268055175 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb744012e40 con 0x7fb7540a19c0 2024-04-19T01:48:59.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb75bbb5640 10 monclient: _renew_subs 2024-04-19T01:48:59.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb75bbb5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:48:59.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb75bbb5640 1 -- 172.21.15.62:0/3268055175 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb75413b970 con 0x7fb7540a19c0 2024-04-19T01:48:59.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb74b7fe640 4 set_mon_vals no callback set 2024-04-19T01:48:59.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb74a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:59.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb74a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:48:59.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb74a7fc640 10 monclient: dump: 2024-04-19T01:48:59.027 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:59.027 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:59.027 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:59.027 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:59.027 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:59.027 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:59.027 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:59.027 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:59.027 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:59.027 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:59.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb75bbb5640 10 monclient: _renew_subs 2024-04-19T01:48:59.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb75bbb5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:48:59.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb75bbb5640 1 -- 172.21.15.62:0/3268055175 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb754070680 con 0x7fb7540a19c0 2024-04-19T01:48:59.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb75bbb5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:48:59.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb75bbb5640 1 -- 172.21.15.62:0/3268055175 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb754072cf0 con 0x7fb7540a19c0 2024-04-19T01:48:59.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb74a7fc640 1 -- 172.21.15.62:0/3268055175 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fb74401c460 con 0x7fb7540a19c0 2024-04-19T01:48:59.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb74a7fc640 1 --2- 172.21.15.62:0/3268055175 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fb738036470 0x7fb738038930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:59.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb74a7fc640 1 -- 172.21.15.62:0/3268055175 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(325..325 src has 115..325) ==== 4346+0+0 (secure 0 0 0) 0x7fb744049140 con 0x7fb7540a19c0 2024-04-19T01:48:59.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb74a7fc640 1 --2- 172.21.15.62:0/3268055175 >> [v2:172.21.15.71:6800/196330660,v1:172.21.15.71:6801/196330660] conn(0x7fb73803c2f0 0x7fb73803e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:59.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb74a7fc640 1 -- 172.21.15.62:0/3268055175 --> [v2:172.21.15.71:6800/196330660,v1:172.21.15.71:6801/196330660] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb73803edc0 con 0x7fb73803c2f0 2024-04-19T01:48:59.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb74a7fc640 1 -- 172.21.15.62:0/3268055175 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_get_version_reply(handle=1 version=325) ==== 24+0+0 (secure 0 0 0) 0x7fb744050050 con 0x7fb7540a19c0 2024-04-19T01:48:59.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb74a7fc640 10 monclient: handle_get_version_reply finishing 1 version 325 2024-04-19T01:48:59.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb759951640 1 --2- 172.21.15.62:0/3268055175 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fb738036470 0x7fb738038930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:59.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb75a152640 1 --2- 172.21.15.62:0/3268055175 >> [v2:172.21.15.71:6800/196330660,v1:172.21.15.71:6801/196330660] conn(0x7fb73803c2f0 0x7fb73803e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:59.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb759951640 10 monclient: get_auth_request con 0x7fb738036470 auth_method 0 2024-04-19T01:48:59.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.021+0000 7fb75a152640 10 monclient: get_auth_request con 0x7fb73803c2f0 auth_method 0 2024-04-19T01:48:59.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.025+0000 7fb759951640 1 --2- 172.21.15.62:0/3268055175 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fb738036470 0x7fb738038930 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb7540a3210 tx=0x7fb7500a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:59.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.029+0000 7fb75a152640 1 --2- 172.21.15.62:0/3268055175 >> [v2:172.21.15.71:6800/196330660,v1:172.21.15.71:6801/196330660] conn(0x7fb73803c2f0 0x7fb73803e710 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:59.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.029+0000 7fb74a7fc640 1 -- 172.21.15.62:0/3268055175 <== osd.5 v2:172.21.15.71:6800/196330660 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb73803edc0 con 0x7fb73803c2f0 2024-04-19T01:48:59.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.041+0000 7fb75bbb5640 1 -- 172.21.15.62:0/3268055175 --> [v2:172.21.15.71:6800/196330660,v1:172.21.15.71:6801/196330660] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fb75404fcf0 con 0x7fb73803c2f0 2024-04-19T01:48:59.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.041+0000 7fb74a7fc640 1 -- 172.21.15.62:0/3268055175 <== osd.5 v2:172.21.15.71:6800/196330660 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fb75404fcf0 con 0x7fb73803c2f0 2024-04-19T01:48:59.039 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-19T01:48:59.039 INFO:teuthology.orchestra.run.smithi062.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T01:48:59.039 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-19T01:48:59.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.041+0000 7fb75bbb5640 1 -- 172.21.15.62:0/3268055175 >> [v2:172.21.15.71:6800/196330660,v1:172.21.15.71:6801/196330660] conn(0x7fb73803c2f0 msgr2=0x7fb73803e710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:59.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.041+0000 7fb75bbb5640 1 --2- 172.21.15.62:0/3268055175 >> [v2:172.21.15.71:6800/196330660,v1:172.21.15.71:6801/196330660] conn(0x7fb73803c2f0 0x7fb73803e710 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:59.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.041+0000 7fb75bbb5640 1 -- 172.21.15.62:0/3268055175 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fb738036470 msgr2=0x7fb738038930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:59.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.041+0000 7fb75bbb5640 1 --2- 172.21.15.62:0/3268055175 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7fb738036470 0x7fb738038930 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb7540a3210 tx=0x7fb7500a7070 comp rx=0 tx=0).stop 2024-04-19T01:48:59.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.041+0000 7fb75bbb5640 10 monclient: shutdown 2024-04-19T01:48:59.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.041+0000 7fb75bbb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:59.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.041+0000 7fb75bbb5640 1 -- 172.21.15.62:0/3268055175 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7540a19c0 msgr2=0x7fb754099690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:59.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.041+0000 7fb75bbb5640 1 --2- 172.21.15.62:0/3268055175 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7fb7540a19c0 0x7fb754099690 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb74400c450 tx=0x7fb74400c910 comp rx=0 tx=0).stop 2024-04-19T01:48:59.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.041+0000 7fb75a152640 1 -- 172.21.15.62:0/3268055175 reap_dead start 2024-04-19T01:48:59.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.041+0000 7fb75bbb5640 1 -- 172.21.15.62:0/3268055175 shutdown_connections 2024-04-19T01:48:59.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.041+0000 7fb75bbb5640 1 -- 172.21.15.62:0/3268055175 >> 172.21.15.62:0/3268055175 conn(0x7fb7540a2190 msgr2=0x7fb7540a4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:59.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.041+0000 7fb75bbb5640 1 -- 172.21.15.62:0/3268055175 shutdown_connections 2024-04-19T01:48:59.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.041+0000 7fb75bbb5640 1 -- 172.21.15.62:0/3268055175 wait complete. 2024-04-19T01:48:59.054 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-19T01:48:59.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b40a7640 10 monclient: get_monmap_and_config 2024-04-19T01:48:59.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b40a7640 10 monclient: build_initial_monmap 2024-04-19T01:48:59.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b40a7640 10 monclient: monmap: 2024-04-19T01:48:59.164 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:59.164 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:59.164 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:48:59.170150+0000 2024-04-19T01:48:59.164 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:48:59.170150+0000 2024-04-19T01:48:59.164 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:59.164 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:59.164 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:48:59.164 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:48:59.165 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:48:59.165 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:48:59.165 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:48:59.165 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:59.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b40a7640 1 Processor -- start 2024-04-19T01:48:59.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b40a7640 1 -- start start 2024-04-19T01:48:59.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b40a7640 10 monclient: init 2024-04-19T01:48:59.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b40a7640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:59.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b40a7640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-04-19T01:48:59.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b40a7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00ac137780 0x7f00ac137b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:59.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b40a7640 10 monclient(hunting): picked mon.noname-b con 0x7f00ac137780 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:59.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b40a7640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f00ac1380a0 0x7f00ac0a45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:59.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b40a7640 10 monclient(hunting): picked mon.noname-c con 0x7f00ac1380a0 addr v2:172.21.15.71:3300/0 2024-04-19T01:48:59.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b40a7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f00ac0a60b0 addr v1:172.21.15.62:6789/0 2024-04-19T01:48:59.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b40a7640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00ac09c2e0 con 0x7f00ac0a60b0 2024-04-19T01:48:59.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b40a7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:59.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b40a7640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f00ac09bd20 con 0x7f00ac1380a0 2024-04-19T01:48:59.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b40a7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:59.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b40a7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f00ac09c000 con 0x7f00ac137780 2024-04-19T01:48:59.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b40a7640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:59.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b40a7640 10 monclient(hunting): authenticate will time out at 1324.989585s 2024-04-19T01:48:59.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b1e43640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00ac137780 0x7f00ac137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:59.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b2644640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f00ac0a60b0 0x7f00ac0a4b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53284/0 (socket says 172.21.15.62:53284) 2024-04-19T01:48:59.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b2644640 1 -- 172.21.15.62:0/4186592203 learned_addr learned my addr 172.21.15.62:0/4186592203 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:48:59.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b1e43640 10 monclient(hunting): get_auth_request con 0x7f00ac137780 auth_method 0 2024-04-19T01:48:59.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b1e43640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:59.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b1e43640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:59.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b1e43640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:59.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b1e43640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:59.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b1e43640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:59.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b1e43640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:59.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b1642640 1 --2- 172.21.15.62:0/4186592203 >> v2:172.21.15.71:3300/0 conn(0x7f00ac1380a0 0x7f00ac0a45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:59.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b1642640 10 monclient(hunting): get_auth_request con 0x7f00ac1380a0 auth_method 0 2024-04-19T01:48:59.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b1642640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:59.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b1642640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:59.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b1642640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:59.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b1e43640 10 monclient(hunting): handle_auth_done global_id 34196 payload 274 2024-04-19T01:48:59.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b1e43640 1 -- 172.21.15.62:0/4186592203 >> v2:172.21.15.71:3300/0 conn(0x7f00ac1380a0 msgr2=0x7f00ac0a45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:59.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b1e43640 1 --2- 172.21.15.62:0/4186592203 >> v2:172.21.15.71:3300/0 conn(0x7f00ac1380a0 0x7f00ac0a45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:59.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b1e43640 1 -- 172.21.15.62:0/4186592203 >> v1:172.21.15.62:6789/0 conn(0x7f00ac0a60b0 legacy=0x7f00ac0a4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:59.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b1e43640 10 monclient: _finish_hunting 0 2024-04-19T01:48:59.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b1e43640 1 monclient: found mon.noname-b 2024-04-19T01:48:59.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b1e43640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:59.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b1e43640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:48:59.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.169+0000 7f00b1e43640 1 -- 172.21.15.62:0/4186592203 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00ac073210 con 0x7f00ac137780 2024-04-19T01:48:59.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.173+0000 7f00b1e43640 1 -- 172.21.15.62:0/4186592203 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00ac137780 msgr2=0x7f00ac137b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:59.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.173+0000 7f00b1e43640 1 --2- 172.21.15.62:0/4186592203 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00ac137780 0x7f00ac137b60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f00a4009d90 tx=0x7f00a401c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fecdc312f250f3fe server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:59.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.173+0000 7f00b0e41640 1 -- 172.21.15.62:0/4186592203 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a40a8610 con 0x7f00ac137780 2024-04-19T01:48:59.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.173+0000 7f00b0e41640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:59.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.173+0000 7f00b0e41640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:48:59.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.173+0000 7f00b0e41640 10 monclient: dump: 2024-04-19T01:48:59.173 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:59.173 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:59.174 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:59.174 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:59.174 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:59.174 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:59.174 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:59.174 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:59.174 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:59.174 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:59.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.173+0000 7f00b0e41640 10 monclient: _finish_auth 0 2024-04-19T01:48:59.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.173+0000 7f00b0e41640 10 monclient: _check_auth_tickets 2024-04-19T01:48:59.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.173+0000 7f00b0e41640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:59.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.173+0000 7f00b0e41640 1 -- 172.21.15.62:0/4186592203 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00a40a87b0 con 0x7f00ac137780 2024-04-19T01:48:59.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.173+0000 7f00b0e41640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:59.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.173+0000 7f00b0e41640 1 -- 172.21.15.62:0/4186592203 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a40a8b80 con 0x7f00ac137780 2024-04-19T01:48:59.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.173+0000 7f00b40a7640 5 monclient: authenticate success, global_id 34196 2024-04-19T01:48:59.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.173+0000 7f00b40a7640 10 monclient: get_monmap_and_config success 2024-04-19T01:48:59.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.173+0000 7f00b40a7640 4 set_mon_vals no callback set 2024-04-19T01:48:59.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.173+0000 7f00b0e41640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:59.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.173+0000 7f00b0e41640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:48:59.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.173+0000 7f00b0e41640 10 monclient: dump: 2024-04-19T01:48:59.175 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:59.175 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:59.175 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:59.176 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:59.176 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:59.176 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:59.176 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:59.176 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:59.176 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:59.176 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:59.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.173+0000 7f00b40a7640 10 monclient: shutdown 2024-04-19T01:48:59.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.173+0000 7f00b40a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:59.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.173+0000 7f00b40a7640 1 -- 172.21.15.62:0/4186592203 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00ac137780 msgr2=0x7f00ac137b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:59.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.173+0000 7f00b40a7640 1 --2- 172.21.15.62:0/4186592203 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00ac137780 0x7f00ac137b60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f00a4009d90 tx=0x7f00a401c650 comp rx=0 tx=0).stop 2024-04-19T01:48:59.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.173+0000 7f00b40a7640 1 -- 172.21.15.62:0/4186592203 shutdown_connections 2024-04-19T01:48:59.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.173+0000 7f00b40a7640 1 --2- 172.21.15.62:0/4186592203 >> v2:172.21.15.71:3300/0 conn(0x7f00ac1380a0 0x7f00ac0a45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:59.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.173+0000 7f00b40a7640 1 --2- 172.21.15.62:0/4186592203 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00ac137780 0x7f00ac137b60 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:59.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.173+0000 7f00b40a7640 1 -- 172.21.15.62:0/4186592203 >> 172.21.15.62:0/4186592203 conn(0x7f00ac0aa1d0 msgr2=0x7f00ac0a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:59.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.173+0000 7f00b40a7640 1 -- 172.21.15.62:0/4186592203 shutdown_connections 2024-04-19T01:48:59.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.173+0000 7f00b40a7640 1 -- 172.21.15.62:0/4186592203 wait complete. 2024-04-19T01:48:59.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.173+0000 7f00b40a7640 10 monclient: build_initial_monmap 2024-04-19T01:48:59.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.173+0000 7f00b40a7640 10 monclient: monmap: 2024-04-19T01:48:59.177 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:59.177 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:48:59.177 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:48:59.177 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:48:59.177 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:59.177 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:59.177 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:48:59.178 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:48:59.178 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:48:59.178 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:59.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b40a7640 1 Processor -- start 2024-04-19T01:48:59.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b40a7640 1 -- start start 2024-04-19T01:48:59.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b40a7640 10 monclient: init 2024-04-19T01:48:59.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b40a7640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:59.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b40a7640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-19T01:48:59.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b40a7640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f00ac0a60b0 0x7f00ac1cf400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:59.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b40a7640 10 monclient(hunting): picked mon.noname-b con 0x7f00ac0a60b0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:48:59.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b40a7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00ac137780 0x7f00ac136310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:59.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b40a7640 10 monclient(hunting): picked mon.noname-a con 0x7f00ac137780 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:48:59.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b40a7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00ac1380a0 0x7f00ac13c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:59.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b40a7640 10 monclient(hunting): picked mon.noname-c con 0x7f00ac1380a0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:59.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b40a7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:59.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b40a7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f00ac142170 con 0x7f00ac137780 2024-04-19T01:48:59.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b40a7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:59.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b40a7640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f00ac0a5430 con 0x7f00ac0a60b0 2024-04-19T01:48:59.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b40a7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:59.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b40a7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f00ac1422f0 con 0x7f00ac1380a0 2024-04-19T01:48:59.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b40a7640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:59.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b40a7640 10 monclient(hunting): authenticate will time out at 1324.996644s 2024-04-19T01:48:59.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b1642640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00ac137780 0x7f00ac136310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:59.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b2644640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00ac1380a0 0x7f00ac13c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:59.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b1642640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00ac137780 0x7f00ac136310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47216/0 (socket says 172.21.15.62:47216) 2024-04-19T01:48:59.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b1642640 1 -- 172.21.15.62:0/2914873615 learned_addr learned my addr 172.21.15.62:0/2914873615 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:59.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b1642640 10 monclient(hunting): get_auth_request con 0x7f00ac137780 auth_method 0 2024-04-19T01:48:59.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b1642640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:59.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b1642640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:59.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b1642640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:59.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b2644640 10 monclient(hunting): get_auth_request con 0x7f00ac1380a0 auth_method 0 2024-04-19T01:48:59.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b2644640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:59.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b2644640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:59.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b2644640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:59.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b1642640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:59.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b1642640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:59.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b1642640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:59.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b2644640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:59.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b2644640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:59.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b2644640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:59.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b1e43640 1 --2- 172.21.15.62:0/2914873615 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f00ac0a60b0 0x7f00ac1cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:59.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b1642640 10 monclient(hunting): handle_auth_done global_id 15363 payload 824 2024-04-19T01:48:59.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b1642640 1 -- 172.21.15.62:0/2914873615 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00ac1380a0 msgr2=0x7f00ac13c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:59.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b1642640 1 --2- 172.21.15.62:0/2914873615 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00ac1380a0 0x7f00ac13c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:59.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b1642640 1 -- 172.21.15.62:0/2914873615 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f00ac0a60b0 msgr2=0x7f00ac1cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:59.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b1642640 1 --2- 172.21.15.62:0/2914873615 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f00ac0a60b0 0x7f00ac1cf400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:59.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b1642640 10 monclient: _finish_hunting 0 2024-04-19T01:48:59.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b1642640 1 monclient: found mon.noname-a 2024-04-19T01:48:59.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b1642640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:59.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b1642640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:48:59.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b1642640 1 -- 172.21.15.62:0/2914873615 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00ac1d0310 con 0x7f00ac137780 2024-04-19T01:48:59.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b1642640 1 -- 172.21.15.62:0/2914873615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00ac137780 msgr2=0x7f00ac136310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:59.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b1642640 1 --2- 172.21.15.62:0/2914873615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00ac137780 0x7f00ac136310 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f00a000d830 tx=0x7f00a000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:59.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f009e7fc640 1 -- 172.21.15.62:0/2914873615 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a000b8d0 con 0x7f00ac137780 2024-04-19T01:48:59.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f009e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:59.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f009e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:48:59.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f009e7fc640 10 monclient: dump: 2024-04-19T01:48:59.184 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:59.184 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:59.184 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:59.184 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:59.184 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:59.184 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:59.184 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:59.184 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:59.185 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:59.185 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:59.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f009e7fc640 10 monclient: _finish_auth 0 2024-04-19T01:48:59.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f009e7fc640 10 monclient: _check_auth_tickets 2024-04-19T01:48:59.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f009e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:59.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f009e7fc640 1 -- 172.21.15.62:0/2914873615 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00a0019070 con 0x7f00ac137780 2024-04-19T01:48:59.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b40a7640 5 monclient: authenticate success, global_id 15363 2024-04-19T01:48:59.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b40a7640 10 monclient: _renew_subs 2024-04-19T01:48:59.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b40a7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:59.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b40a7640 1 -- 172.21.15.62:0/2914873615 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00ac0a3420 con 0x7f00ac137780 2024-04-19T01:48:59.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f009e7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:59.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f009e7fc640 1 -- 172.21.15.62:0/2914873615 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a000bc70 con 0x7f00ac137780 2024-04-19T01:48:59.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f009e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:59.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f009e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:48:59.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f009f7fe640 4 set_mon_vals no callback set 2024-04-19T01:48:59.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f009e7fc640 10 monclient: dump: 2024-04-19T01:48:59.186 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:59.186 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:59.186 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:59.186 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:59.186 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:59.186 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:59.186 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:59.186 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:59.186 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:59.186 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:59.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b40a7640 10 monclient: _renew_subs 2024-04-19T01:48:59.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b40a7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:59.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.177+0000 7f00b40a7640 1 -- 172.21.15.62:0/2914873615 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00ac120ca0 con 0x7f00ac137780 2024-04-19T01:48:59.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.181+0000 7f00b40a7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:59.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.181+0000 7f00b40a7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:59.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.181+0000 7f00b40a7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:48:59.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.181+0000 7f00b40a7640 1 -- 172.21.15.62:0/2914873615 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00ac068ab0 con 0x7f00ac137780 2024-04-19T01:48:59.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.181+0000 7f009e7fc640 1 -- 172.21.15.62:0/2914873615 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f00a001e460 con 0x7f00ac137780 2024-04-19T01:48:59.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.181+0000 7f009e7fc640 1 --2- 172.21.15.62:0/2914873615 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0090036420 0x7f00900388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:59.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.181+0000 7f009e7fc640 1 -- 172.21.15.62:0/2914873615 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(325..325 src has 115..325) ==== 4346+0+0 (secure 0 0 0) 0x7f00a0049810 con 0x7f00ac137780 2024-04-19T01:48:59.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.181+0000 7f00b1e43640 1 --2- 172.21.15.62:0/2914873615 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0090036420 0x7f00900388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:59.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.181+0000 7f00b1e43640 10 monclient: get_auth_request con 0x7f0090036420 auth_method 0 2024-04-19T01:48:59.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.185+0000 7f00b1e43640 1 --2- 172.21.15.62:0/2914873615 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0090036420 0x7f00900388e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f00a40ac000 tx=0x7f00a40a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:59.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.185+0000 7f009e7fc640 1 -- 172.21.15.62:0/2914873615 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f00ac1d0310 con 0x7f00ac137780 2024-04-19T01:48:59.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.185+0000 7f009e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:59.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.185+0000 7f009e7fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:48:59.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.345+0000 7f00b40a7640 1 -- 172.21.15.62:0/2914873615 --> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f00ac13ec00 con 0x7f0090036420 2024-04-19T01:48:59.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.345+0000 7f009e7fc640 1 -- 172.21.15.62:0/2914873615 <== mgr.4108 v2:172.21.15.71:6824/2133533992 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f00ac13ec00 con 0x7f0090036420 2024-04-19T01:48:59.341 INFO:teuthology.orchestra.run.smithi062.stderr:instructed osd(s) 5 to deep-scrub 2024-04-19T01:48:59.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.349+0000 7f00b40a7640 1 -- 172.21.15.62:0/2914873615 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0090036420 msgr2=0x7f00900388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:59.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.349+0000 7f00b40a7640 1 --2- 172.21.15.62:0/2914873615 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0090036420 0x7f00900388e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f00a40ac000 tx=0x7f00a40a7000 comp rx=0 tx=0).stop 2024-04-19T01:48:59.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.349+0000 7f00b40a7640 10 monclient: shutdown 2024-04-19T01:48:59.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.349+0000 7f00b40a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:59.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.349+0000 7f00b40a7640 1 -- 172.21.15.62:0/2914873615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00ac137780 msgr2=0x7f00ac136310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:59.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.349+0000 7f00b40a7640 1 --2- 172.21.15.62:0/2914873615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00ac137780 0x7f00ac136310 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f00a000d830 tx=0x7f00a000dcf0 comp rx=0 tx=0).stop 2024-04-19T01:48:59.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.349+0000 7f00b40a7640 1 -- 172.21.15.62:0/2914873615 shutdown_connections 2024-04-19T01:48:59.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.349+0000 7f00b40a7640 1 --2- 172.21.15.62:0/2914873615 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f0090036420 0x7f00900388e0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:59.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.349+0000 7f00b40a7640 1 --2- 172.21.15.62:0/2914873615 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f00ac1380a0 0x7f00ac13c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:59.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.349+0000 7f00b40a7640 1 --2- 172.21.15.62:0/2914873615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00ac137780 0x7f00ac136310 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:59.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.349+0000 7f00b40a7640 1 --2- 172.21.15.62:0/2914873615 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f00ac0a60b0 0x7f00ac1cf400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:59.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.349+0000 7f00b40a7640 1 -- 172.21.15.62:0/2914873615 >> 172.21.15.62:0/2914873615 conn(0x7f00ac0aa1d0 msgr2=0x7f00ac0a79d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:59.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.349+0000 7f00b40a7640 1 -- 172.21.15.62:0/2914873615 shutdown_connections 2024-04-19T01:48:59.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.349+0000 7f00b40a7640 1 -- 172.21.15.62:0/2914873615 wait complete. 2024-04-19T01:48:59.359 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:48:59.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.473+0000 7faf48df7640 10 monclient: get_monmap_and_config 2024-04-19T01:48:59.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.473+0000 7faf48df7640 10 monclient: build_initial_monmap 2024-04-19T01:48:59.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.473+0000 7faf48df7640 10 monclient: monmap: 2024-04-19T01:48:59.470 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:59.471 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:59.471 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:48:59.476795+0000 2024-04-19T01:48:59.471 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:48:59.476795+0000 2024-04-19T01:48:59.471 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:59.471 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:59.471 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:48:59.471 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:48:59.471 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:48:59.471 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:48:59.471 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:48:59.471 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:59.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.473+0000 7faf48df7640 1 Processor -- start 2024-04-19T01:48:59.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.473+0000 7faf48df7640 1 -- start start 2024-04-19T01:48:59.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.473+0000 7faf48df7640 10 monclient: init 2024-04-19T01:48:59.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.473+0000 7faf48df7640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:59.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf48df7640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-04-19T01:48:59.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf48df7640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7faf44137780 0x7faf44137b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:59.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf48df7640 10 monclient(hunting): picked mon.noname-a con 0x7faf44137780 addr v2:172.21.15.62:3300/0 2024-04-19T01:48:59.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf48df7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faf441380a0 addr v1:172.21.15.62:6789/0 2024-04-19T01:48:59.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf48df7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faf440a60b0 0x7faf440a4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:59.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf48df7640 10 monclient(hunting): picked mon.noname-b con 0x7faf440a60b0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:59.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf48df7640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf4409c2e0 con 0x7faf441380a0 2024-04-19T01:48:59.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf48df7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:59.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf48df7640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7faf4409bd20 con 0x7faf44137780 2024-04-19T01:48:59.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf48df7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:59.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf48df7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7faf4409c000 con 0x7faf440a60b0 2024-04-19T01:48:59.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf48df7640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:59.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf48df7640 10 monclient(hunting): authenticate will time out at 1325.296080s 2024-04-19T01:48:59.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf42d9d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faf440a60b0 0x7faf440a4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:59.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf4259c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7faf44137780 0x7faf44137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:59.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf42d9d640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faf440a60b0 0x7faf440a4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:40076/0 (socket says 172.21.15.62:40076) 2024-04-19T01:48:59.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf41d9b640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7faf441380a0 0x7faf440a45d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53288/0 (socket says 172.21.15.62:53288) 2024-04-19T01:48:59.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf4259c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7faf44137780 0x7faf44137b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47218/0 (socket says 172.21.15.62:47218) 2024-04-19T01:48:59.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf42d9d640 1 -- 172.21.15.62:0/3177785225 learned_addr learned my addr 172.21.15.62:0/3177785225 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:59.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf42d9d640 10 monclient(hunting): get_auth_request con 0x7faf440a60b0 auth_method 0 2024-04-19T01:48:59.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf42d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:59.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf42d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:59.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf42d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:59.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf4259c640 10 monclient(hunting): get_auth_request con 0x7faf44137780 auth_method 0 2024-04-19T01:48:59.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf4259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:59.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf4259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:59.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf4259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:59.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf42d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:59.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf42d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:59.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf42d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:59.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf4259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:59.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf4259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:59.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf4259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:59.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf42d9d640 10 monclient(hunting): handle_auth_done global_id 34202 payload 274 2024-04-19T01:48:59.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf42d9d640 1 -- 172.21.15.62:0/3177785225 >> v2:172.21.15.62:3300/0 conn(0x7faf44137780 msgr2=0x7faf44137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:59.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf42d9d640 1 --2- 172.21.15.62:0/3177785225 >> v2:172.21.15.62:3300/0 conn(0x7faf44137780 0x7faf44137b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:59.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf42d9d640 1 -- 172.21.15.62:0/3177785225 >> v1:172.21.15.62:6789/0 conn(0x7faf441380a0 legacy=0x7faf440a45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:48:59.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf42d9d640 10 monclient: _finish_hunting 0 2024-04-19T01:48:59.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf42d9d640 1 monclient: found mon.noname-b 2024-04-19T01:48:59.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf42d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:59.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf42d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.62:3301/0 2024-04-19T01:48:59.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf42d9d640 1 -- 172.21.15.62:0/3177785225 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf44073210 con 0x7faf440a60b0 2024-04-19T01:48:59.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.477+0000 7faf4259c640 1 --2- 172.21.15.62:0/3177785225 >> v2:172.21.15.62:3300/0 conn(0x7faf44137780 0x7faf44137b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:48:59.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf42d9d640 1 -- 172.21.15.62:0/3177785225 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faf440a60b0 msgr2=0x7faf440a4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:59.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf42d9d640 1 --2- 172.21.15.62:0/3177785225 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faf440a60b0 0x7faf440a4ce0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7faf3c009870 tx=0x7faf3c01c240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4aaa8bbbdaceeb88 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:59.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4159a640 1 -- 172.21.15.62:0/3177785225 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf3c0b0070 con 0x7faf440a60b0 2024-04-19T01:48:59.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:59.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:48:59.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4159a640 10 monclient: dump: 2024-04-19T01:48:59.482 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:59.482 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:59.482 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:59.482 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:59.482 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:59.482 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:59.482 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:59.482 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:59.482 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:59.483 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:59.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4159a640 10 monclient: _finish_auth 0 2024-04-19T01:48:59.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4159a640 10 monclient: _check_auth_tickets 2024-04-19T01:48:59.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:59.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4159a640 1 -- 172.21.15.62:0/3177785225 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf3c0ab600 con 0x7faf440a60b0 2024-04-19T01:48:59.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4159a640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:59.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 5 monclient: authenticate success, global_id 34202 2024-04-19T01:48:59.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 10 monclient: get_monmap_and_config success 2024-04-19T01:48:59.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 4 set_mon_vals no callback set 2024-04-19T01:48:59.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4159a640 1 -- 172.21.15.62:0/3177785225 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf3c0ab9d0 con 0x7faf440a60b0 2024-04-19T01:48:59.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:59.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:48:59.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4159a640 10 monclient: dump: 2024-04-19T01:48:59.484 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:59.484 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:59.484 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:59.484 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:59.484 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:59.484 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:59.484 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:59.484 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:59.484 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:59.484 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:59.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 10 monclient: shutdown 2024-04-19T01:48:59.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:59.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 1 -- 172.21.15.62:0/3177785225 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faf440a60b0 msgr2=0x7faf440a4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:59.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 1 --2- 172.21.15.62:0/3177785225 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faf440a60b0 0x7faf440a4ce0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7faf3c009870 tx=0x7faf3c01c240 comp rx=0 tx=0).stop 2024-04-19T01:48:59.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 1 -- 172.21.15.62:0/3177785225 shutdown_connections 2024-04-19T01:48:59.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 1 --2- 172.21.15.62:0/3177785225 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faf440a60b0 0x7faf440a4ce0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:59.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 1 --2- 172.21.15.62:0/3177785225 >> v2:172.21.15.62:3300/0 conn(0x7faf44137780 0x7faf44137b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:59.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 1 -- 172.21.15.62:0/3177785225 >> 172.21.15.62:0/3177785225 conn(0x7faf440aa1d0 msgr2=0x7faf440a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:59.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 1 -- 172.21.15.62:0/3177785225 shutdown_connections 2024-04-19T01:48:59.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 1 -- 172.21.15.62:0/3177785225 wait complete. 2024-04-19T01:48:59.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 10 monclient: build_initial_monmap 2024-04-19T01:48:59.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 10 monclient: monmap: 2024-04-19T01:48:59.485 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:48:59.486 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:48:59.486 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:48:59.486 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:48:59.486 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:48:59.486 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:59.486 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:48:59.486 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:48:59.486 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:48:59.486 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:59.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 1 Processor -- start 2024-04-19T01:48:59.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 1 -- start start 2024-04-19T01:48:59.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 10 monclient: init 2024-04-19T01:48:59.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 10 monclient: _reopen_session rank -1 2024-04-19T01:48:59.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-19T01:48:59.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faf440a60b0 0x7faf441cf580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:59.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 10 monclient(hunting): picked mon.noname-c con 0x7faf440a60b0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:48:59.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf44137780 0x7faf441365d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:59.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 10 monclient(hunting): picked mon.noname-a con 0x7faf44137780 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:48:59.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf441380a0 0x7faf4413c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:59.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 10 monclient(hunting): picked mon.noname-b con 0x7faf441380a0 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:48:59.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:59.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7faf44142170 con 0x7faf44137780 2024-04-19T01:48:59.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:59.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7faf440a5430 con 0x7faf441380a0 2024-04-19T01:48:59.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:48:59.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7faf441422f0 con 0x7faf440a60b0 2024-04-19T01:48:59.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 10 monclient(hunting): _renew_subs 2024-04-19T01:48:59.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf48df7640 10 monclient(hunting): authenticate will time out at 1325.302901s 2024-04-19T01:48:59.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4259c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faf440a60b0 0x7faf441cf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:59.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf41d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf44137780 0x7faf441365d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:59.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4259c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faf440a60b0 0x7faf441cf580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:40092/0 (socket says 172.21.15.62:40092) 2024-04-19T01:48:59.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4259c640 1 -- 172.21.15.62:0/2773540672 learned_addr learned my addr 172.21.15.62:0/2773540672 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:48:59.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4259c640 10 monclient(hunting): get_auth_request con 0x7faf440a60b0 auth_method 0 2024-04-19T01:48:59.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:59.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:59.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:59.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf41d9b640 10 monclient(hunting): get_auth_request con 0x7faf44137780 auth_method 0 2024-04-19T01:48:59.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf41d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:48:59.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf41d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:48:59.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf41d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:48:59.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:59.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:59.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:59.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf41d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:48:59.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf41d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:48:59.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf41d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:48:59.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf42d9d640 1 --2- 172.21.15.62:0/2773540672 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf441380a0 0x7faf4413c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:59.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4259c640 10 monclient(hunting): handle_auth_done global_id 34205 payload 824 2024-04-19T01:48:59.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4259c640 1 -- 172.21.15.62:0/2773540672 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf441380a0 msgr2=0x7faf4413c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:59.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4259c640 1 --2- 172.21.15.62:0/2773540672 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf441380a0 0x7faf4413c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:59.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4259c640 1 -- 172.21.15.62:0/2773540672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf44137780 msgr2=0x7faf441365d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:59.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4259c640 1 --2- 172.21.15.62:0/2773540672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf44137780 0x7faf441365d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:59.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4259c640 10 monclient: _finish_hunting 0 2024-04-19T01:48:59.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4259c640 1 monclient: found mon.noname-c 2024-04-19T01:48:59.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:48:59.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:48:59.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf4259c640 1 -- 172.21.15.62:0/2773540672 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf440a3420 con 0x7faf440a60b0 2024-04-19T01:48:59.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf42d9d640 10 monclient: get_auth_request con 0x7faf441380a0 auth_method 0 2024-04-19T01:48:59.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf42d9d640 1 --2- 172.21.15.62:0/2773540672 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf441380a0 0x7faf4413c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:48:59.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.481+0000 7faf41d9b640 1 --2- 172.21.15.62:0/2773540672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf44137780 0x7faf441365d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:48:59.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.485+0000 7faf4259c640 1 -- 172.21.15.62:0/2773540672 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faf440a60b0 msgr2=0x7faf441cf580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:48:59.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.485+0000 7faf4259c640 1 --2- 172.21.15.62:0/2773540672 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faf440a60b0 0x7faf441cf580 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7faf3400eed0 tx=0x7faf3400b480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:59.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.485+0000 7faf3affd640 1 -- 172.21.15.62:0/2773540672 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf34018020 con 0x7faf440a60b0 2024-04-19T01:48:59.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.485+0000 7faf3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:59.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.485+0000 7faf3affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:48:59.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.485+0000 7faf3affd640 10 monclient: dump: 2024-04-19T01:48:59.491 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:59.491 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:59.491 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:59.491 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:59.491 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:59.491 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:59.491 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:59.491 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:59.491 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:59.491 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:59.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.485+0000 7faf3affd640 10 monclient: _finish_auth 0 2024-04-19T01:48:59.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.485+0000 7faf3affd640 10 monclient: _check_auth_tickets 2024-04-19T01:48:59.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.485+0000 7faf3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:48:59.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.485+0000 7faf3affd640 1 -- 172.21.15.62:0/2773540672 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf34014810 con 0x7faf440a60b0 2024-04-19T01:48:59.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.485+0000 7faf3affd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:48:59.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.485+0000 7faf3affd640 1 -- 172.21.15.62:0/2773540672 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf3400be60 con 0x7faf440a60b0 2024-04-19T01:48:59.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.485+0000 7faf3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:48:59.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.485+0000 7faf3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:48:59.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.485+0000 7faf3affd640 10 monclient: dump: 2024-04-19T01:48:59.493 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:48:59.493 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:48:59.493 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:59.493 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:48:59.493 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:48:59.493 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:48:59.494 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:48:59.494 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:48:59.494 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:48:59.494 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:48:59.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.485+0000 7faf48df7640 5 monclient: authenticate success, global_id 34205 2024-04-19T01:48:59.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.485+0000 7faf3bfff640 4 set_mon_vals no callback set 2024-04-19T01:48:59.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.485+0000 7faf48df7640 10 monclient: _renew_subs 2024-04-19T01:48:59.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.485+0000 7faf48df7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:48:59.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.485+0000 7faf48df7640 1 -- 172.21.15.62:0/2773540672 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf440a22a0 con 0x7faf440a60b0 2024-04-19T01:48:59.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.485+0000 7faf48df7640 10 monclient: _renew_subs 2024-04-19T01:48:59.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.485+0000 7faf48df7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:48:59.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.485+0000 7faf48df7640 1 -- 172.21.15.62:0/2773540672 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faf4413df40 con 0x7faf440a60b0 2024-04-19T01:48:59.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.485+0000 7faf48df7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:59.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.485+0000 7faf48df7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:59.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.485+0000 7faf48df7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:48:59.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.485+0000 7faf3affd640 1 -- 172.21.15.62:0/2773540672 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7faf3401b020 con 0x7faf440a60b0 2024-04-19T01:48:59.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.485+0000 7faf48df7640 1 -- 172.21.15.62:0/2773540672 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf44068ab0 con 0x7faf440a60b0 2024-04-19T01:48:59.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.485+0000 7faf3affd640 1 --2- 172.21.15.62:0/2773540672 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7faf24036420 0x7faf240388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:48:59.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.485+0000 7faf3affd640 1 -- 172.21.15.62:0/2773540672 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(325..325 src has 115..325) ==== 4346+0+0 (secure 0 0 0) 0x7faf34049c90 con 0x7faf440a60b0 2024-04-19T01:48:59.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.485+0000 7faf41d9b640 1 --2- 172.21.15.62:0/2773540672 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7faf24036420 0x7faf240388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:48:59.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.485+0000 7faf41d9b640 10 monclient: get_auth_request con 0x7faf24036420 auth_method 0 2024-04-19T01:48:59.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.489+0000 7faf41d9b640 1 --2- 172.21.15.62:0/2773540672 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7faf24036420 0x7faf240388e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7faf4413db40 tx=0x7faf2c007e70 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:48:59.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.489+0000 7faf3affd640 1 -- 172.21.15.62:0/2773540672 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7faf34050080 con 0x7faf440a60b0 2024-04-19T01:48:59.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.489+0000 7faf3affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:48:59.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.489+0000 7faf3affd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:48:59.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.645+0000 7faf48df7640 1 -- 172.21.15.62:0/2773540672 --> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7faf44068710 con 0x7faf24036420 2024-04-19T01:48:59.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.645+0000 7faf3affd640 1 -- 172.21.15.62:0/2773540672 <== mgr.4108 v2:172.21.15.71:6824/2133533992 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33777 (secure 0 0 0) 0x7faf44068710 con 0x7faf24036420 2024-04-19T01:48:59.644 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:48:59.644 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-04-19T01:48:59.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.649+0000 7faf48df7640 1 -- 172.21.15.62:0/2773540672 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7faf24036420 msgr2=0x7faf240388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:59.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.649+0000 7faf48df7640 1 --2- 172.21.15.62:0/2773540672 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7faf24036420 0x7faf240388e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7faf4413db40 tx=0x7faf2c007e70 comp rx=0 tx=0).stop 2024-04-19T01:48:59.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.649+0000 7faf48df7640 10 monclient: shutdown 2024-04-19T01:48:59.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.649+0000 7faf48df7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:48:59.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.649+0000 7faf48df7640 1 -- 172.21.15.62:0/2773540672 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faf440a60b0 msgr2=0x7faf441cf580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:48:59.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.649+0000 7faf48df7640 1 --2- 172.21.15.62:0/2773540672 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faf440a60b0 0x7faf441cf580 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7faf3400eed0 tx=0x7faf3400b480 comp rx=0 tx=0).stop 2024-04-19T01:48:59.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.649+0000 7faf48df7640 1 -- 172.21.15.62:0/2773540672 shutdown_connections 2024-04-19T01:48:59.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.649+0000 7faf48df7640 1 --2- 172.21.15.62:0/2773540672 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf441380a0 0x7faf4413c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:59.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.649+0000 7faf48df7640 1 --2- 172.21.15.62:0/2773540672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf44137780 0x7faf441365d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:59.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.649+0000 7faf48df7640 1 --2- 172.21.15.62:0/2773540672 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7faf24036420 0x7faf240388e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:59.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.649+0000 7faf48df7640 1 --2- 172.21.15.62:0/2773540672 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7faf440a60b0 0x7faf441cf580 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:48:59.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.649+0000 7faf48df7640 1 -- 172.21.15.62:0/2773540672 >> 172.21.15.62:0/2773540672 conn(0x7faf440aa1d0 msgr2=0x7faf440a79d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:48:59.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.649+0000 7faf48df7640 1 -- 172.21.15.62:0/2773540672 shutdown_connections 2024-04-19T01:48:59.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:48:59.649+0000 7faf48df7640 1 -- 172.21.15.62:0/2773540672 wait complete. 2024-04-19T01:48:59.661 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":520,"stamp":"2024-04-19T01:48:58.818182+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":101,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":193244,"kb_used_data":18632,"kb_used_omap":9,"kb_used_meta":174518,"kb_avail":566037796,"statfs":{"total":579820584960,"available":579622703104,"internally_reserved":0,"allocated":19079168,"data_stored":7581674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":178707104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003039"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":621,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:47.760922+0000","last_change":"2024-04-19T01:42:24.162228+0000","last_active":"2024-04-19T01:48:47.760922+0000","last_peered":"2024-04-19T01:48:47.760922+0000","last_clean":"2024-04-19T01:48:47.760922+0000","last_became_active":"2024-04-19T01:42:24.161645+0000","last_became_peered":"2024-04-19T01:42:24.161645+0000","last_unstale":"2024-04-19T01:48:47.760922+0000","last_undegraded":"2024-04-19T01:48:47.760922+0000","last_fullsized":"2024-04-19T01:48:47.760922+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T06:14:08.038010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":618,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:47.491774+0000","last_change":"2024-04-19T01:42:24.132804+0000","last_active":"2024-04-19T01:48:47.491774+0000","last_peered":"2024-04-19T01:48:47.491774+0000","last_clean":"2024-04-19T01:48:47.491774+0000","last_became_active":"2024-04-19T01:42:24.131937+0000","last_became_peered":"2024-04-19T01:42:24.131937+0000","last_unstale":"2024-04-19T01:48:47.491774+0000","last_undegraded":"2024-04-19T01:48:47.491774+0000","last_fullsized":"2024-04-19T01:48:47.491774+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T13:40:26.146488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":621,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:47.760787+0000","last_change":"2024-04-19T01:42:24.135590+0000","last_active":"2024-04-19T01:48:47.760787+0000","last_peered":"2024-04-19T01:48:47.760787+0000","last_clean":"2024-04-19T01:48:47.760787+0000","last_became_active":"2024-04-19T01:42:24.134918+0000","last_became_peered":"2024-04-19T01:42:24.134918+0000","last_unstale":"2024-04-19T01:48:47.760787+0000","last_undegraded":"2024-04-19T01:48:47.760787+0000","last_fullsized":"2024-04-19T01:48:47.760787+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T11:52:05.398825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":618,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:47.491620+0000","last_change":"2024-04-19T01:42:24.132575+0000","last_active":"2024-04-19T01:48:47.491620+0000","last_peered":"2024-04-19T01:48:47.491620+0000","last_clean":"2024-04-19T01:48:47.491620+0000","last_became_active":"2024-04-19T01:42:24.131894+0000","last_became_peered":"2024-04-19T01:42:24.131894+0000","last_unstale":"2024-04-19T01:48:47.491620+0000","last_undegraded":"2024-04-19T01:48:47.491620+0000","last_fullsized":"2024-04-19T01:48:47.491620+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T07:32:38.880998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":615,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:46.659613+0000","last_change":"2024-04-19T01:42:24.145699+0000","last_active":"2024-04-19T01:48:46.659613+0000","last_peered":"2024-04-19T01:48:46.659613+0000","last_clean":"2024-04-19T01:48:46.659613+0000","last_became_active":"2024-04-19T01:42:24.144746+0000","last_became_peered":"2024-04-19T01:42:24.144746+0000","last_unstale":"2024-04-19T01:48:46.659613+0000","last_undegraded":"2024-04-19T01:48:46.659613+0000","last_fullsized":"2024-04-19T01:48:46.659613+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:21:11.400175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":617,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:47.762428+0000","last_change":"2024-04-19T01:42:24.142754+0000","last_active":"2024-04-19T01:48:47.762428+0000","last_peered":"2024-04-19T01:48:47.762428+0000","last_clean":"2024-04-19T01:48:47.762428+0000","last_became_active":"2024-04-19T01:42:24.142309+0000","last_became_peered":"2024-04-19T01:42:24.142309+0000","last_unstale":"2024-04-19T01:48:47.762428+0000","last_undegraded":"2024-04-19T01:48:47.762428+0000","last_fullsized":"2024-04-19T01:48:47.762428+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:40:28.826490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":621,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:47.760787+0000","last_change":"2024-04-19T01:42:24.134484+0000","last_active":"2024-04-19T01:48:47.760787+0000","last_peered":"2024-04-19T01:48:47.760787+0000","last_clean":"2024-04-19T01:48:47.760787+0000","last_became_active":"2024-04-19T01:42:24.133282+0000","last_became_peered":"2024-04-19T01:42:24.133282+0000","last_unstale":"2024-04-19T01:48:47.760787+0000","last_undegraded":"2024-04-19T01:48:47.760787+0000","last_fullsized":"2024-04-19T01:48:47.760787+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T13:07:26.373782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":615,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:46.659609+0000","last_change":"2024-04-19T01:42:24.146550+0000","last_active":"2024-04-19T01:48:46.659609+0000","last_peered":"2024-04-19T01:48:46.659609+0000","last_clean":"2024-04-19T01:48:46.659609+0000","last_became_active":"2024-04-19T01:42:24.145667+0000","last_became_peered":"2024-04-19T01:42:24.145667+0000","last_unstale":"2024-04-19T01:48:46.659609+0000","last_undegraded":"2024-04-19T01:48:46.659609+0000","last_fullsized":"2024-04-19T01:48:46.659609+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:23.115035+0000","last_clean_scrub_stamp":"2024-04-19T01:42:23.115035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T10:07:16.240297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":776,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:47.760789+0000","last_change":"2024-04-19T01:42:22.132251+0000","last_active":"2024-04-19T01:48:47.760789+0000","last_peered":"2024-04-19T01:48:47.760789+0000","last_clean":"2024-04-19T01:48:47.760789+0000","last_became_active":"2024-04-19T01:42:22.131432+0000","last_became_peered":"2024-04-19T01:42:22.131432+0000","last_unstale":"2024-04-19T01:48:47.760789+0000","last_undegraded":"2024-04-19T01:48:47.760789+0000","last_fullsized":"2024-04-19T01:48:47.760789+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:42:21.102542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:42:21.102542+0000","last_clean_scrub_stamp":"2024-04-19T01:42:21.102542+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T06:47:12.046147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":101,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":101,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607634,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34112,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337728,"statfs":{"total":96636764160,"available":96601833472,"internally_reserved":0,"allocated":3002368,"data_stored":1087303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640337,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34112,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337728,"statfs":{"total":96636764160,"available":96601833472,"internally_reserved":0,"allocated":3002368,"data_stored":1087303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640338,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30476,"kb_used_data":3452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341364,"statfs":{"total":96636764160,"available":96605556736,"internally_reserved":0,"allocated":3534848,"data_stored":1616231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640338,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29956,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341884,"statfs":{"total":96636764160,"available":96606089216,"internally_reserved":0,"allocated":3002368,"data_stored":1087303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640338,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29952,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341888,"statfs":{"total":96636764160,"available":96606093312,"internally_reserved":0,"allocated":3002368,"data_stored":1087303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640339,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34636,"kb_used_data":3452,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337204,"statfs":{"total":96636764160,"available":96601296896,"internally_reserved":0,"allocated":3534848,"data_stored":1616231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:48:59.664 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-19T01:42:23.115035+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=42, tm_sec=23, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=48, tm_sec=55, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:48:59.665 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-19T01:42:23.115035+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=42, tm_sec=23, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=48, tm_sec=55, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:48:59.665 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-19T01:42:23.115035+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=42, tm_sec=23, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=48, tm_sec=55, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:48:59.665 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-19T01:42:23.115035+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=42, tm_sec=23, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=48, tm_sec=55, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:48:59.665 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-19T01:42:23.115035+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=42, tm_sec=23, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=48, tm_sec=55, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:48:59.665 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-19T01:42:23.115035+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=42, tm_sec=23, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=48, tm_sec=55, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:48:59.665 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-19T01:42:23.115035+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=42, tm_sec=23, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=48, tm_sec=55, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:48:59.665 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-19T01:42:23.115035+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=42, tm_sec=23, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=48, tm_sec=55, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:48:59.665 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-19T01:42:21.102542+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=42, tm_sec=21, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=48, tm_sec=55, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:48:59.665 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-19T01:49:19.666 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:49:19.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.805+0000 7f5d84bfb640 10 monclient: get_monmap_and_config 2024-04-19T01:49:19.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.805+0000 7f5d84bfb640 10 monclient: build_initial_monmap 2024-04-19T01:49:19.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.805+0000 7f5d84bfb640 10 monclient: monmap: 2024-04-19T01:49:19.803 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:49:19.803 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:49:19.803 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:49:19.808835+0000 2024-04-19T01:49:19.803 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:49:19.808835+0000 2024-04-19T01:49:19.803 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:49:19.803 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:49:19.803 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:49:19.804 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:49:19.804 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:49:19.804 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:49:19.804 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:49:19.804 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:49:19.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d84bfb640 1 Processor -- start 2024-04-19T01:49:19.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d84bfb640 1 -- start start 2024-04-19T01:49:19.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d84bfb640 10 monclient: init 2024-04-19T01:49:19.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d84bfb640 10 monclient: _reopen_session rank -1 2024-04-19T01:49:19.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d84bfb640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-04-19T01:49:19.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d84bfb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5d80137780 addr v1:172.21.15.62:6789/0 2024-04-19T01:49:19.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d84bfb640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5d800a60b0 0x7f5d8013c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:49:19.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d84bfb640 10 monclient(hunting): picked mon.noname-a con 0x7f5d800a60b0 addr v2:172.21.15.62:3300/0 2024-04-19T01:49:19.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d84bfb640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f5d8013cc60 0x7f5d8013f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:49:19.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d84bfb640 10 monclient(hunting): picked mon.noname-c con 0x7f5d8013cc60 addr v2:172.21.15.71:3300/0 2024-04-19T01:49:19.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d84bfb640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d8009c2e0 con 0x7f5d80137780 2024-04-19T01:49:19.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d84bfb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:49:19.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d84bfb640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f5d8009bd20 con 0x7f5d800a60b0 2024-04-19T01:49:19.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d84bfb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:49:19.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d84bfb640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 -- 0x7f5d8009c000 con 0x7f5d8013cc60 2024-04-19T01:49:19.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d84bfb640 10 monclient(hunting): _renew_subs 2024-04-19T01:49:19.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d84bfb640 10 monclient(hunting): authenticate will time out at 1345.629108s 2024-04-19T01:49:19.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d77fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5d800a60b0 0x7f5d8013c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:49:19.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d7e59c640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f5d80137780 0x7f5d80137b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:55912/0 (socket says 172.21.15.62:55912) 2024-04-19T01:49:19.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d77fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5d800a60b0 0x7f5d8013c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60738/0 (socket says 172.21.15.62:60738) 2024-04-19T01:49:19.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d7e59c640 1 -- 172.21.15.62:0/866543773 learned_addr learned my addr 172.21.15.62:0/866543773 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-19T01:49:19.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d77fff640 10 monclient(hunting): get_auth_request con 0x7f5d800a60b0 auth_method 0 2024-04-19T01:49:19.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:49:19.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d77fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:49:19.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d77fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:49:19.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:49:19.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:49:19.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:49:19.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d7ed9d640 1 --2- 172.21.15.62:0/866543773 >> v2:172.21.15.71:3300/0 conn(0x7f5d8013cc60 0x7f5d8013f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:49:19.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d7ed9d640 10 monclient(hunting): get_auth_request con 0x7f5d8013cc60 auth_method 0 2024-04-19T01:49:19.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d7ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:49:19.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d7ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:49:19.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d7ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:49:19.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d77fff640 10 monclient(hunting): handle_auth_done global_id 15372 payload 274 2024-04-19T01:49:19.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d77fff640 1 -- 172.21.15.62:0/866543773 >> v2:172.21.15.71:3300/0 conn(0x7f5d8013cc60 msgr2=0x7f5d8013f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:49:19.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d77fff640 1 --2- 172.21.15.62:0/866543773 >> v2:172.21.15.71:3300/0 conn(0x7f5d8013cc60 0x7f5d8013f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:49:19.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d77fff640 1 -- 172.21.15.62:0/866543773 >> v1:172.21.15.62:6789/0 conn(0x7f5d80137780 legacy=0x7f5d80137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:49:19.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d77fff640 10 monclient: _finish_hunting 0 2024-04-19T01:49:19.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d77fff640 1 monclient: found mon.noname-a 2024-04-19T01:49:19.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:49:19.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d77fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:49:19.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.809+0000 7f5d77fff640 1 -- 172.21.15.62:0/866543773 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d80138270 con 0x7f5d800a60b0 2024-04-19T01:49:19.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.813+0000 7f5d77fff640 1 -- 172.21.15.62:0/866543773 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d800a60b0 msgr2=0x7f5d8013c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:49:19.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.813+0000 7f5d77fff640 1 --2- 172.21.15.62:0/866543773 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d800a60b0 0x7f5d8013c720 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f5d8009bc20 tx=0x7f5d6801c1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=556cdcfe668f7b4a server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:49:19.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.813+0000 7f5d7dd9b640 1 -- 172.21.15.62:0/866543773 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d680b0070 con 0x7f5d800a60b0 2024-04-19T01:49:19.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.813+0000 7f5d7dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:49:19.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.813+0000 7f5d7dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:49:19.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.813+0000 7f5d7dd9b640 10 monclient: dump: 2024-04-19T01:49:19.815 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:49:19.815 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:49:19.815 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:49:19.815 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:49:19.816 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:49:19.816 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:49:19.816 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:49:19.816 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:49:19.816 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:49:19.816 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:49:19.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.813+0000 7f5d7dd9b640 10 monclient: _finish_auth 0 2024-04-19T01:49:19.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.813+0000 7f5d7dd9b640 10 monclient: _check_auth_tickets 2024-04-19T01:49:19.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.813+0000 7f5d7dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:49:19.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.813+0000 7f5d7dd9b640 1 -- 172.21.15.62:0/866543773 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d680ab600 con 0x7f5d800a60b0 2024-04-19T01:49:19.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.813+0000 7f5d84bfb640 5 monclient: authenticate success, global_id 15372 2024-04-19T01:49:19.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.813+0000 7f5d84bfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:49:19.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.813+0000 7f5d7dd9b640 10 monclient: handle_config config(0 keys) 2024-04-19T01:49:19.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.813+0000 7f5d84bfb640 10 monclient: get_monmap_and_config success 2024-04-19T01:49:19.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.813+0000 7f5d84bfb640 4 set_mon_vals no callback set 2024-04-19T01:49:19.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.813+0000 7f5d7dd9b640 1 -- 172.21.15.62:0/866543773 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d680ab9d0 con 0x7f5d800a60b0 2024-04-19T01:49:19.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.813+0000 7f5d7dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:49:19.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.813+0000 7f5d7dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:49:19.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.813+0000 7f5d7dd9b640 10 monclient: dump: 2024-04-19T01:49:19.817 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:49:19.817 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:49:19.817 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:49:19.817 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:49:19.817 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:49:19.817 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:49:19.818 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:49:19.818 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:49:19.818 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:49:19.818 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:49:19.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.813+0000 7f5d84bfb640 10 monclient: shutdown 2024-04-19T01:49:19.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.813+0000 7f5d84bfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:49:19.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.813+0000 7f5d84bfb640 1 -- 172.21.15.62:0/866543773 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d800a60b0 msgr2=0x7f5d8013c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:49:19.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.813+0000 7f5d84bfb640 1 --2- 172.21.15.62:0/866543773 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d800a60b0 0x7f5d8013c720 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f5d8009bc20 tx=0x7f5d6801c1f0 comp rx=0 tx=0).stop 2024-04-19T01:49:19.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.813+0000 7f5d84bfb640 1 -- 172.21.15.62:0/866543773 shutdown_connections 2024-04-19T01:49:19.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.813+0000 7f5d84bfb640 1 --2- 172.21.15.62:0/866543773 >> v2:172.21.15.71:3300/0 conn(0x7f5d8013cc60 0x7f5d8013f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:49:19.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.813+0000 7f5d84bfb640 1 --2- 172.21.15.62:0/866543773 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d800a60b0 0x7f5d8013c720 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:49:19.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.813+0000 7f5d84bfb640 1 -- 172.21.15.62:0/866543773 >> 172.21.15.62:0/866543773 conn(0x7f5d800aa1d0 msgr2=0x7f5d800a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:49:19.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.813+0000 7f5d84bfb640 1 -- 172.21.15.62:0/866543773 shutdown_connections 2024-04-19T01:49:19.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.813+0000 7f5d84bfb640 1 -- 172.21.15.62:0/866543773 wait complete. 2024-04-19T01:49:19.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d84bfb640 10 monclient: build_initial_monmap 2024-04-19T01:49:19.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d84bfb640 10 monclient: monmap: 2024-04-19T01:49:19.819 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:49:19.819 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:49:19.819 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:49:19.819 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:49:19.819 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:49:19.819 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:49:19.819 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:49:19.819 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:49:19.819 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:49:19.820 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:49:19.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d84bfb640 1 Processor -- start 2024-04-19T01:49:19.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d84bfb640 1 -- start start 2024-04-19T01:49:19.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d84bfb640 10 monclient: init 2024-04-19T01:49:19.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d84bfb640 10 monclient: _reopen_session rank -1 2024-04-19T01:49:19.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d84bfb640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-19T01:49:19.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d84bfb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d800a60b0 0x7f5d801cf450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:49:19.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d84bfb640 10 monclient(hunting): picked mon.noname-c con 0x7f5d800a60b0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:49:19.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d84bfb640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d80137780 0x7f5d80136430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:49:19.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d84bfb640 10 monclient(hunting): picked mon.noname-a con 0x7f5d80137780 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:49:19.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d84bfb640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d8013cc60 0x7f5d8009f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:49:19.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d84bfb640 10 monclient(hunting): picked mon.noname-b con 0x7f5d8013cc60 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:49:19.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d84bfb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:49:19.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d84bfb640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5d80142140 con 0x7f5d80137780 2024-04-19T01:49:19.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d84bfb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:49:19.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d84bfb640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5d80141fc0 con 0x7f5d8013cc60 2024-04-19T01:49:19.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d84bfb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:49:19.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d84bfb640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f5d801422c0 con 0x7f5d800a60b0 2024-04-19T01:49:19.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d84bfb640 10 monclient(hunting): _renew_subs 2024-04-19T01:49:19.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d84bfb640 10 monclient(hunting): authenticate will time out at 1345.637326s 2024-04-19T01:49:19.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d7e59c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d800a60b0 0x7f5d801cf450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:49:19.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d77fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d80137780 0x7f5d80136430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:49:19.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d7e59c640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d800a60b0 0x7f5d801cf450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3301/0 says I am v2:172.21.15.62:34450/0 (socket says 172.21.15.62:34450) 2024-04-19T01:49:19.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d7e59c640 1 -- 172.21.15.62:0/2706114594 learned_addr learned my addr 172.21.15.62:0/2706114594 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:49:19.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d7e59c640 10 monclient(hunting): get_auth_request con 0x7f5d800a60b0 auth_method 0 2024-04-19T01:49:19.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d7e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:49:19.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d7e59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:49:19.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d7e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:49:19.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d77fff640 10 monclient(hunting): get_auth_request con 0x7f5d80137780 auth_method 0 2024-04-19T01:49:19.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:49:19.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d77fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:49:19.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d77fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:49:19.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d7e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:49:19.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d7e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:49:19.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d7e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:49:19.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:49:19.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:49:19.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:49:19.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d7ed9d640 1 --2- 172.21.15.62:0/2706114594 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d8013cc60 0x7f5d8009f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:49:19.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d7ed9d640 10 monclient(hunting): get_auth_request con 0x7f5d8013cc60 auth_method 0 2024-04-19T01:49:19.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d7ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:49:19.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d7ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:49:19.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d7ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:49:19.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d7e59c640 10 monclient(hunting): handle_auth_done global_id 34208 payload 824 2024-04-19T01:49:19.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d7e59c640 1 -- 172.21.15.62:0/2706114594 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d8013cc60 msgr2=0x7f5d8009f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:49:19.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d7e59c640 1 --2- 172.21.15.62:0/2706114594 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d8013cc60 0x7f5d8009f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:49:19.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d7e59c640 1 -- 172.21.15.62:0/2706114594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d80137780 msgr2=0x7f5d80136430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:49:19.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d7e59c640 1 --2- 172.21.15.62:0/2706114594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d80137780 0x7f5d80136430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:49:19.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d7e59c640 10 monclient: _finish_hunting 0 2024-04-19T01:49:19.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d7e59c640 1 monclient: found mon.noname-c 2024-04-19T01:49:19.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d7e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:49:19.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d7e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.62:3301/0 2024-04-19T01:49:19.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d7e59c640 1 -- 172.21.15.62:0/2706114594 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d800a1510 con 0x7f5d800a60b0 2024-04-19T01:49:19.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.817+0000 7f5d77fff640 1 --2- 172.21.15.62:0/2706114594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d80137780 0x7f5d80136430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:49:19.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.821+0000 7f5d7e59c640 1 -- 172.21.15.62:0/2706114594 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d800a60b0 msgr2=0x7f5d801cf450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:49:19.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.821+0000 7f5d7e59c640 1 --2- 172.21.15.62:0/2706114594 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d800a60b0 0x7f5d801cf450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5d7000c4a0 tx=0x7f5d7000c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:49:19.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.821+0000 7f5d76ffd640 1 -- 172.21.15.62:0/2706114594 <== mon.2 v2:172.21.15.62:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d70017070 con 0x7f5d800a60b0 2024-04-19T01:49:19.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.821+0000 7f5d76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:49:19.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.821+0000 7f5d76ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:49:19.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.821+0000 7f5d76ffd640 10 monclient: dump: 2024-04-19T01:49:19.827 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:49:19.827 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:49:19.827 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:49:19.827 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:49:19.827 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:49:19.827 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:49:19.827 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:49:19.827 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:49:19.827 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:49:19.827 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:49:19.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.821+0000 7f5d76ffd640 10 monclient: _finish_auth 0 2024-04-19T01:49:19.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.821+0000 7f5d76ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:49:19.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.821+0000 7f5d76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:49:19.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.821+0000 7f5d76ffd640 1 -- 172.21.15.62:0/2706114594 <== mon.2 v2:172.21.15.62:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d70007070 con 0x7f5d800a60b0 2024-04-19T01:49:19.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.821+0000 7f5d84bfb640 5 monclient: authenticate success, global_id 34208 2024-04-19T01:49:19.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.821+0000 7f5d84bfb640 10 monclient: _renew_subs 2024-04-19T01:49:19.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.821+0000 7f5d84bfb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:49:19.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.821+0000 7f5d84bfb640 1 -- 172.21.15.62:0/2706114594 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d800a3420 con 0x7f5d800a60b0 2024-04-19T01:49:19.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.821+0000 7f5d76ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:49:19.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.821+0000 7f5d76ffd640 1 -- 172.21.15.62:0/2706114594 <== mon.2 v2:172.21.15.62:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d70007460 con 0x7f5d800a60b0 2024-04-19T01:49:19.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.821+0000 7f5d76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:49:19.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.821+0000 7f5d76ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:49:19.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.821+0000 7f5d76ffd640 10 monclient: dump: 2024-04-19T01:49:19.828 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:49:19.828 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:49:19.829 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:49:19.829 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:49:19.829 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:49:19.829 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:49:19.829 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:49:19.829 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:49:19.829 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:49:19.829 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:49:19.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.821+0000 7f5d7d59a640 4 set_mon_vals no callback set 2024-04-19T01:49:19.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.821+0000 7f5d84bfb640 10 monclient: _renew_subs 2024-04-19T01:49:19.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.821+0000 7f5d84bfb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:49:19.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.821+0000 7f5d84bfb640 1 -- 172.21.15.62:0/2706114594 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d80120520 con 0x7f5d800a60b0 2024-04-19T01:49:19.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.821+0000 7f5d84bfb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:49:19.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.821+0000 7f5d84bfb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:49:19.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.821+0000 7f5d84bfb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.62:3301/0 2024-04-19T01:49:19.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.821+0000 7f5d84bfb640 1 -- 172.21.15.62:0/2706114594 --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d80068ab0 con 0x7f5d800a60b0 2024-04-19T01:49:19.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.821+0000 7f5d76ffd640 1 -- 172.21.15.62:0/2706114594 <== mon.2 v2:172.21.15.62:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f5d7001a070 con 0x7f5d800a60b0 2024-04-19T01:49:19.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.821+0000 7f5d76ffd640 1 --2- 172.21.15.62:0/2706114594 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5d60036420 0x7f5d600388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:49:19.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.821+0000 7f5d76ffd640 1 -- 172.21.15.62:0/2706114594 <== mon.2 v2:172.21.15.62:3301/0 5 ==== osd_map(325..325 src has 115..325) ==== 4346+0+0 (secure 0 0 0) 0x7f5d700487b0 con 0x7f5d800a60b0 2024-04-19T01:49:19.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.821+0000 7f5d77fff640 1 --2- 172.21.15.62:0/2706114594 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5d60036420 0x7f5d600388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:49:19.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.821+0000 7f5d77fff640 10 monclient: get_auth_request con 0x7f5d60036420 auth_method 0 2024-04-19T01:49:19.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.825+0000 7f5d77fff640 1 --2- 172.21.15.62:0/2706114594 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5d60036420 0x7f5d600388e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5d680ab000 tx=0x7f5d680a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:49:19.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.825+0000 7f5d76ffd640 1 -- 172.21.15.62:0/2706114594 <== mon.2 v2:172.21.15.62:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5d70010300 con 0x7f5d800a60b0 2024-04-19T01:49:19.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.825+0000 7f5d76ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:49:19.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.825+0000 7f5d76ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:49:19.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.981+0000 7f5d84bfb640 1 -- 172.21.15.62:0/2706114594 --> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5d80120900 con 0x7f5d60036420 2024-04-19T01:49:19.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.985+0000 7f5d76ffd640 1 -- 172.21.15.62:0/2706114594 <== mgr.4108 v2:172.21.15.71:6824/2133533992 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33786 (secure 0 0 0) 0x7f5d80120900 con 0x7f5d60036420 2024-04-19T01:49:19.983 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-19T01:49:19.983 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-04-19T01:49:19.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.989+0000 7f5d84bfb640 1 -- 172.21.15.62:0/2706114594 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5d60036420 msgr2=0x7f5d600388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:49:19.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.989+0000 7f5d84bfb640 1 --2- 172.21.15.62:0/2706114594 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5d60036420 0x7f5d600388e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5d680ab000 tx=0x7f5d680a6000 comp rx=0 tx=0).stop 2024-04-19T01:49:19.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.989+0000 7f5d84bfb640 10 monclient: shutdown 2024-04-19T01:49:19.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.989+0000 7f5d84bfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:49:19.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.989+0000 7f5d84bfb640 1 -- 172.21.15.62:0/2706114594 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d800a60b0 msgr2=0x7f5d801cf450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:49:19.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.989+0000 7f5d84bfb640 1 --2- 172.21.15.62:0/2706114594 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d800a60b0 0x7f5d801cf450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5d7000c4a0 tx=0x7f5d7000c960 comp rx=0 tx=0).stop 2024-04-19T01:49:19.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.989+0000 7f5d84bfb640 1 -- 172.21.15.62:0/2706114594 shutdown_connections 2024-04-19T01:49:19.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.989+0000 7f5d84bfb640 1 --2- 172.21.15.62:0/2706114594 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d8013cc60 0x7f5d8009f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:49:19.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.989+0000 7f5d84bfb640 1 --2- 172.21.15.62:0/2706114594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d80137780 0x7f5d80136430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:49:19.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.989+0000 7f5d84bfb640 1 --2- 172.21.15.62:0/2706114594 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f5d60036420 0x7f5d600388e0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:49:19.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.989+0000 7f5d84bfb640 1 --2- 172.21.15.62:0/2706114594 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f5d800a60b0 0x7f5d801cf450 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5d7000c4a0 tx=0x7f5d7000c960 comp rx=0 tx=0).stop 2024-04-19T01:49:19.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.989+0000 7f5d84bfb640 1 -- 172.21.15.62:0/2706114594 >> 172.21.15.62:0/2706114594 conn(0x7f5d800aa1d0 msgr2=0x7f5d8013dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:49:19.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.989+0000 7f5d84bfb640 1 -- 172.21.15.62:0/2706114594 shutdown_connections 2024-04-19T01:49:19.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:19.989+0000 7f5d84bfb640 1 -- 172.21.15.62:0/2706114594 wait complete. 2024-04-19T01:49:20.001 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":530,"stamp":"2024-04-19T01:49:18.823257+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":101,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":193212,"kb_used_data":18632,"kb_used_omap":9,"kb_used_meta":174518,"kb_avail":566037828,"statfs":{"total":579820584960,"available":579622735872,"internally_reserved":0,"allocated":19079168,"data_stored":7581674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":178707104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003083"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":630,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:49:01.979462+0000","last_change":"2024-04-19T01:49:01.979519+0000","last_active":"2024-04-19T01:49:01.979462+0000","last_peered":"2024-04-19T01:49:01.979462+0000","last_clean":"2024-04-19T01:49:01.979462+0000","last_became_active":"2024-04-19T01:42:24.161645+0000","last_became_peered":"2024-04-19T01:42:24.161645+0000","last_unstale":"2024-04-19T01:49:01.979462+0000","last_undegraded":"2024-04-19T01:49:01.979462+0000","last_fullsized":"2024-04-19T01:49:01.979462+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:49:01.979270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:49:01.979270+0000","last_clean_scrub_stamp":"2024-04-19T01:49:01.979270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:12:44.199200+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":627,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:58.644358+0000","last_change":"2024-04-19T01:48:58.644385+0000","last_active":"2024-04-19T01:48:58.644358+0000","last_peered":"2024-04-19T01:48:58.644358+0000","last_clean":"2024-04-19T01:48:58.644358+0000","last_became_active":"2024-04-19T01:42:24.131937+0000","last_became_peered":"2024-04-19T01:42:24.131937+0000","last_unstale":"2024-04-19T01:48:58.644358+0000","last_undegraded":"2024-04-19T01:48:58.644358+0000","last_fullsized":"2024-04-19T01:48:58.644358+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:48:58.644258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:48:58.644258+0000","last_clean_scrub_stamp":"2024-04-19T01:48:58.644258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T13:18:38.118646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":630,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:49:00.995218+0000","last_change":"2024-04-19T01:49:00.995342+0000","last_active":"2024-04-19T01:49:00.995218+0000","last_peered":"2024-04-19T01:49:00.995218+0000","last_clean":"2024-04-19T01:49:00.995218+0000","last_became_active":"2024-04-19T01:42:24.134918+0000","last_became_peered":"2024-04-19T01:42:24.134918+0000","last_unstale":"2024-04-19T01:49:00.995218+0000","last_undegraded":"2024-04-19T01:49:00.995218+0000","last_fullsized":"2024-04-19T01:49:00.995218+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:49:00.995016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:49:00.995016+0000","last_clean_scrub_stamp":"2024-04-19T01:49:00.995016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T05:33:07.302737+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":627,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:57.598956+0000","last_change":"2024-04-19T01:48:57.598977+0000","last_active":"2024-04-19T01:48:57.598956+0000","last_peered":"2024-04-19T01:48:57.598956+0000","last_clean":"2024-04-19T01:48:57.598956+0000","last_became_active":"2024-04-19T01:42:24.131894+0000","last_became_peered":"2024-04-19T01:42:24.131894+0000","last_unstale":"2024-04-19T01:48:57.598956+0000","last_undegraded":"2024-04-19T01:48:57.598956+0000","last_fullsized":"2024-04-19T01:48:57.598956+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:48:57.598869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:48:57.598869+0000","last_clean_scrub_stamp":"2024-04-19T01:48:57.598869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T03:11:07.621376+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":624,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:59.635345+0000","last_change":"2024-04-19T01:48:59.635405+0000","last_active":"2024-04-19T01:48:59.635345+0000","last_peered":"2024-04-19T01:48:59.635345+0000","last_clean":"2024-04-19T01:48:59.635345+0000","last_became_active":"2024-04-19T01:42:24.144746+0000","last_became_peered":"2024-04-19T01:42:24.144746+0000","last_unstale":"2024-04-19T01:48:59.635345+0000","last_undegraded":"2024-04-19T01:48:59.635345+0000","last_fullsized":"2024-04-19T01:48:59.635345+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:48:59.635170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:48:59.635170+0000","last_clean_scrub_stamp":"2024-04-19T01:48:59.635170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T13:13:14.535337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":626,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:58.796670+0000","last_change":"2024-04-19T01:48:58.796798+0000","last_active":"2024-04-19T01:48:58.796670+0000","last_peered":"2024-04-19T01:48:58.796670+0000","last_clean":"2024-04-19T01:48:58.796670+0000","last_became_active":"2024-04-19T01:42:24.142309+0000","last_became_peered":"2024-04-19T01:42:24.142309+0000","last_unstale":"2024-04-19T01:48:58.796670+0000","last_undegraded":"2024-04-19T01:48:58.796670+0000","last_fullsized":"2024-04-19T01:48:58.796670+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:48:58.796482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:48:58.796482+0000","last_clean_scrub_stamp":"2024-04-19T01:48:58.796482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T10:27:07.009647+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":630,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:49:00.022300+0000","last_change":"2024-04-19T01:49:00.022352+0000","last_active":"2024-04-19T01:49:00.022300+0000","last_peered":"2024-04-19T01:49:00.022300+0000","last_clean":"2024-04-19T01:49:00.022300+0000","last_became_active":"2024-04-19T01:42:24.133282+0000","last_became_peered":"2024-04-19T01:42:24.133282+0000","last_unstale":"2024-04-19T01:49:00.022300+0000","last_undegraded":"2024-04-19T01:49:00.022300+0000","last_fullsized":"2024-04-19T01:49:00.022300+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:49:00.022134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:49:00.022134+0000","last_clean_scrub_stamp":"2024-04-19T01:49:00.022134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:41:45.381489+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":624,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:49:00.669341+0000","last_change":"2024-04-19T01:49:00.669475+0000","last_active":"2024-04-19T01:49:00.669341+0000","last_peered":"2024-04-19T01:49:00.669341+0000","last_clean":"2024-04-19T01:49:00.669341+0000","last_became_active":"2024-04-19T01:42:24.145667+0000","last_became_peered":"2024-04-19T01:42:24.145667+0000","last_unstale":"2024-04-19T01:49:00.669341+0000","last_undegraded":"2024-04-19T01:49:00.669341+0000","last_fullsized":"2024-04-19T01:49:00.669341+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:49:00.669137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:49:00.669137+0000","last_clean_scrub_stamp":"2024-04-19T01:49:00.669137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T04:25:54.407218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"325'119","reported_seq":787,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-04-19T01:48:59.055259+0000","last_change":"2024-04-19T01:48:59.055341+0000","last_active":"2024-04-19T01:48:59.055259+0000","last_peered":"2024-04-19T01:48:59.055259+0000","last_clean":"2024-04-19T01:48:59.055259+0000","last_became_active":"2024-04-19T01:42:22.131432+0000","last_became_peered":"2024-04-19T01:42:22.131432+0000","last_unstale":"2024-04-19T01:48:59.055259+0000","last_undegraded":"2024-04-19T01:48:59.055259+0000","last_fullsized":"2024-04-19T01:48:59.055259+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'119","last_scrub_stamp":"2024-04-19T01:48:59.055092+0000","last_deep_scrub":"325'119","last_deep_scrub_stamp":"2024-04-19T01:48:59.055092+0000","last_clean_scrub_stamp":"2024-04-19T01:48:59.055092+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T08:07:36.528321+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":101,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":101,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":12,"seq":51539607638,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34112,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337728,"statfs":{"total":96636764160,"available":96601833472,"internally_reserved":0,"allocated":3002368,"data_stored":1087303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640341,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34108,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337732,"statfs":{"total":96636764160,"available":96601837568,"internally_reserved":0,"allocated":3002368,"data_stored":1087303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640342,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30468,"kb_used_data":3452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341372,"statfs":{"total":96636764160,"available":96605564928,"internally_reserved":0,"allocated":3534848,"data_stored":1616231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640342,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29948,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341892,"statfs":{"total":96636764160,"available":96606097408,"internally_reserved":0,"allocated":3002368,"data_stored":1087303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640342,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29948,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341892,"statfs":{"total":96636764160,"available":96606097408,"internally_reserved":0,"allocated":3002368,"data_stored":1087303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640343,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34628,"kb_used_data":3452,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337212,"statfs":{"total":96636764160,"available":96601305088,"internally_reserved":0,"allocated":3534848,"data_stored":1616231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:49:20.002 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-19T01:49:20.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f4153b2f640 10 monclient: get_monmap_and_config 2024-04-19T01:49:20.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f4153b2f640 10 monclient: build_initial_monmap 2024-04-19T01:49:20.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f4153b2f640 10 monclient: monmap: 2024-04-19T01:49:20.118 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:49:20.118 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:49:20.118 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:49:20.123715+0000 2024-04-19T01:49:20.118 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:49:20.123715+0000 2024-04-19T01:49:20.118 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:49:20.118 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:49:20.118 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.62:6789/0 mon.noname-a-legacy 2024-04-19T01:49:20.118 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.71:6789/0 mon.noname-c-legacy 2024-04-19T01:49:20.118 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-b 2024-04-19T01:49:20.118 INFO:teuthology.orchestra.run.smithi062.stderr:3: v2:172.21.15.62:3300/0 mon.noname-a 2024-04-19T01:49:20.118 INFO:teuthology.orchestra.run.smithi062.stderr:4: v2:172.21.15.71:3300/0 mon.noname-c 2024-04-19T01:49:20.118 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:49:20.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f4153b2f640 1 Processor -- start 2024-04-19T01:49:20.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f4153b2f640 1 -- start start 2024-04-19T01:49:20.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f4153b2f640 10 monclient: init 2024-04-19T01:49:20.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f4153b2f640 10 monclient: _reopen_session rank -1 2024-04-19T01:49:20.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f4153b2f640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-04-19T01:49:20.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f4153b2f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f414c09d040 addr v1:172.21.15.71:6789/0 2024-04-19T01:49:20.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f4153b2f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f414c0a2ef0 0x7f414c1391d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:49:20.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f4153b2f640 10 monclient(hunting): picked mon.noname-a con 0x7f414c0a2ef0 addr v2:172.21.15.62:3300/0 2024-04-19T01:49:20.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f4153b2f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f414c139710 addr v1:172.21.15.62:6789/0 2024-04-19T01:49:20.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f4153b2f640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f414c09d4d0 con 0x7f414c139710 2024-04-19T01:49:20.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f4153b2f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f414c09d960 con 0x7f414c09d040 2024-04-19T01:49:20.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f4153b2f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:49:20.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f4153b2f640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f414c0a4ef0 con 0x7f414c0a2ef0 2024-04-19T01:49:20.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f4153b2f640 10 monclient(hunting): _renew_subs 2024-04-19T01:49:20.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f4153b2f640 10 monclient(hunting): authenticate will time out at 1345.943297s 2024-04-19T01:49:20.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f41510ca640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f414c0a2ef0 0x7f414c1391d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:49:20.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f41510ca640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f414c0a2ef0 0x7f414c1391d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60762/0 (socket says 172.21.15.62:60762) 2024-04-19T01:49:20.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f41520cc640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f414c139710 0x7f414c13bb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:55926/0 (socket says 172.21.15.62:55926) 2024-04-19T01:49:20.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f41510ca640 1 -- 172.21.15.62:0/2098226749 learned_addr learned my addr 172.21.15.62:0/2098226749 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:49:20.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f41510ca640 10 monclient(hunting): get_auth_request con 0x7f414c0a2ef0 auth_method 0 2024-04-19T01:49:20.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f41510ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:49:20.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f41510ca640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:49:20.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f41510ca640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:49:20.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f41510ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:49:20.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f41510ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:49:20.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f41510ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:49:20.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f41510ca640 10 monclient(hunting): handle_auth_done global_id 15378 payload 274 2024-04-19T01:49:20.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f41510ca640 1 -- 172.21.15.62:0/2098226749 >> v1:172.21.15.71:6789/0 conn(0x7f414c09d040 legacy=0x7f414c0a47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:49:20.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f41510ca640 1 -- 172.21.15.62:0/2098226749 >> v1:172.21.15.62:6789/0 conn(0x7f414c139710 legacy=0x7f414c13bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:49:20.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f41510ca640 10 monclient: _finish_hunting 0 2024-04-19T01:49:20.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f41510ca640 1 monclient: found mon.noname-a 2024-04-19T01:49:20.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f41510ca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:49:20.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f41510ca640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:49:20.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.121+0000 7f41510ca640 1 -- 172.21.15.62:0/2098226749 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f414c09de60 con 0x7f414c0a2ef0 2024-04-19T01:49:20.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.125+0000 7f41510ca640 1 -- 172.21.15.62:0/2098226749 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f414c0a2ef0 msgr2=0x7f414c1391d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:49:20.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.125+0000 7f41510ca640 1 --2- 172.21.15.62:0/2098226749 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f414c0a2ef0 0x7f414c1391d0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f414c0762a0 tx=0x7f413c01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21ca460cc6514b12 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:49:20.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41508c9640 1 -- 172.21.15.62:0/2098226749 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413c0b0070 con 0x7f414c0a2ef0 2024-04-19T01:49:20.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41508c9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:49:20.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41508c9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:49:20.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41508c9640 10 monclient: dump: 2024-04-19T01:49:20.124 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:49:20.125 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:49:20.125 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:49:20.125 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:49:20.125 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:49:20.125 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:49:20.125 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:49:20.125 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:49:20.125 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:49:20.125 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:49:20.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41508c9640 10 monclient: _finish_auth 0 2024-04-19T01:49:20.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41508c9640 10 monclient: _check_auth_tickets 2024-04-19T01:49:20.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41508c9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:49:20.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41508c9640 1 -- 172.21.15.62:0/2098226749 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f413c0ab600 con 0x7f414c0a2ef0 2024-04-19T01:49:20.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41508c9640 10 monclient: handle_config config(0 keys) 2024-04-19T01:49:20.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41508c9640 1 -- 172.21.15.62:0/2098226749 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413c0ab9d0 con 0x7f414c0a2ef0 2024-04-19T01:49:20.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 5 monclient: authenticate success, global_id 15378 2024-04-19T01:49:20.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 10 monclient: get_monmap_and_config success 2024-04-19T01:49:20.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 4 set_mon_vals no callback set 2024-04-19T01:49:20.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41508c9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:49:20.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41508c9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:49:20.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41508c9640 10 monclient: dump: 2024-04-19T01:49:20.126 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:49:20.126 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:49:20.126 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:49:20.126 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:49:20.126 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:49:20.126 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:49:20.126 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:49:20.126 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:49:20.126 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:49:20.127 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:49:20.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 10 monclient: shutdown 2024-04-19T01:49:20.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:49:20.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 1 -- 172.21.15.62:0/2098226749 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f414c0a2ef0 msgr2=0x7f414c1391d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:49:20.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 1 --2- 172.21.15.62:0/2098226749 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f414c0a2ef0 0x7f414c1391d0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f414c0762a0 tx=0x7f413c01c400 comp rx=0 tx=0).stop 2024-04-19T01:49:20.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 1 -- 172.21.15.62:0/2098226749 shutdown_connections 2024-04-19T01:49:20.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 1 --2- 172.21.15.62:0/2098226749 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f414c0a2ef0 0x7f414c1391d0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:49:20.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 1 -- 172.21.15.62:0/2098226749 >> 172.21.15.62:0/2098226749 conn(0x7f414c0a23d0 msgr2=0x7f414c0a27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:49:20.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 1 -- 172.21.15.62:0/2098226749 shutdown_connections 2024-04-19T01:49:20.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 1 -- 172.21.15.62:0/2098226749 wait complete. 2024-04-19T01:49:20.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 10 monclient: build_initial_monmap 2024-04-19T01:49:20.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 10 monclient: monmap: 2024-04-19T01:49:20.131 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-04-19T01:49:20.131 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:49:20.131 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-04-19T01:49:20.131 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-04-19T01:49:20.131 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-04-19T01:49:20.132 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:49:20.132 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.noname-a 2024-04-19T01:49:20.132 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.noname-b 2024-04-19T01:49:20.132 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.noname-c 2024-04-19T01:49:20.132 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:49:20.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 1 Processor -- start 2024-04-19T01:49:20.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 1 -- start start 2024-04-19T01:49:20.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 10 monclient: init 2024-04-19T01:49:20.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 10 monclient: _reopen_session rank -1 2024-04-19T01:49:20.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-19T01:49:20.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f414c0a2ef0 0x7f414c071060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:49:20.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 10 monclient(hunting): picked mon.noname-c con 0x7f414c0a2ef0 addr [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-04-19T01:49:20.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f414c139710 0x7f414c134600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:49:20.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 10 monclient(hunting): picked mon.noname-b con 0x7f414c139710 addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-19T01:49:20.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f414c134b40 0x7f414c1cc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:49:20.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 10 monclient(hunting): picked mon.noname-a con 0x7f414c134b40 addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] 2024-04-19T01:49:20.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:49:20.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f414c13fdd0 con 0x7f414c134b40 2024-04-19T01:49:20.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:49:20.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f414c13fc50 con 0x7f414c139710 2024-04-19T01:49:20.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:49:20.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 1 -- --> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] -- mon_getmap magic: 0 -- 0x7f414c13ff50 con 0x7f414c0a2ef0 2024-04-19T01:49:20.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 10 monclient(hunting): _renew_subs 2024-04-19T01:49:20.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f4153b2f640 10 monclient(hunting): authenticate will time out at 1345.950468s 2024-04-19T01:49:20.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41520cc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f414c134b40 0x7f414c1cc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:49:20.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41518cb640 1 --2- >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f414c0a2ef0 0x7f414c071060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:49:20.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41520cc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f414c134b40 0x7f414c1cc1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60766/0 (socket says 172.21.15.62:60766) 2024-04-19T01:49:20.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41520cc640 1 -- 172.21.15.62:0/1717291854 learned_addr learned my addr 172.21.15.62:0/1717291854 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-19T01:49:20.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41520cc640 10 monclient(hunting): get_auth_request con 0x7f414c134b40 auth_method 0 2024-04-19T01:49:20.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41520cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:49:20.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41520cc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:49:20.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41520cc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:49:20.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41518cb640 10 monclient(hunting): get_auth_request con 0x7f414c0a2ef0 auth_method 0 2024-04-19T01:49:20.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41518cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:49:20.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41518cb640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:49:20.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41518cb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:49:20.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41520cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:49:20.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41520cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:49:20.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41520cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:49:20.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41518cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:49:20.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41518cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:49:20.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41518cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:49:20.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41510ca640 1 --2- 172.21.15.62:0/1717291854 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f414c139710 0x7f414c134600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:49:20.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41510ca640 10 monclient(hunting): get_auth_request con 0x7f414c139710 auth_method 0 2024-04-19T01:49:20.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41510ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:49:20.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41510ca640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:49:20.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41510ca640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:49:20.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41520cc640 10 monclient(hunting): handle_auth_done global_id 15381 payload 824 2024-04-19T01:49:20.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41520cc640 1 -- 172.21.15.62:0/1717291854 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f414c0a2ef0 msgr2=0x7f414c071060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:49:20.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41520cc640 1 --2- 172.21.15.62:0/1717291854 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f414c0a2ef0 0x7f414c071060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:49:20.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41520cc640 1 -- 172.21.15.62:0/1717291854 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f414c139710 msgr2=0x7f414c134600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:49:20.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41520cc640 1 --2- 172.21.15.62:0/1717291854 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f414c139710 0x7f414c134600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:49:20.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41520cc640 10 monclient: _finish_hunting 0 2024-04-19T01:49:20.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41520cc640 1 monclient: found mon.noname-a 2024-04-19T01:49:20.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41520cc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:49:20.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41520cc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.62:3300/0 2024-04-19T01:49:20.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.129+0000 7f41520cc640 1 -- 172.21.15.62:0/1717291854 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f414c13b1c0 con 0x7f414c134b40 2024-04-19T01:49:20.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.133+0000 7f41520cc640 1 -- 172.21.15.62:0/1717291854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f414c134b40 msgr2=0x7f414c1cc1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-19T01:49:20.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.133+0000 7f41520cc640 1 --2- 172.21.15.62:0/1717291854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f414c134b40 0x7f414c1cc1e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f414400c710 tx=0x7f414400cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:49:20.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.133+0000 7f41427fc640 1 -- 172.21.15.62:0/1717291854 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4144017070 con 0x7f414c134b40 2024-04-19T01:49:20.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.133+0000 7f41427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:49:20.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.133+0000 7f41427fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:49:20.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.133+0000 7f41427fc640 10 monclient: dump: 2024-04-19T01:49:20.136 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:49:20.136 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:49:20.136 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:49:20.136 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:49:20.136 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:49:20.136 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:49:20.136 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:49:20.136 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:49:20.137 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:49:20.137 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:49:20.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.133+0000 7f41427fc640 10 monclient: _finish_auth 0 2024-04-19T01:49:20.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.133+0000 7f41427fc640 10 monclient: _check_auth_tickets 2024-04-19T01:49:20.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.133+0000 7f41427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:49:20.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.133+0000 7f41427fc640 1 -- 172.21.15.62:0/1717291854 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f414400a860 con 0x7f414c134b40 2024-04-19T01:49:20.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.133+0000 7f4153b2f640 5 monclient: authenticate success, global_id 15381 2024-04-19T01:49:20.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.133+0000 7f4153b2f640 10 monclient: _renew_subs 2024-04-19T01:49:20.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.133+0000 7f4153b2f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:49:20.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.133+0000 7f4153b2f640 1 -- 172.21.15.62:0/1717291854 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f414c120710 con 0x7f414c134b40 2024-04-19T01:49:20.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.133+0000 7f41427fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:49:20.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.133+0000 7f41427fc640 1 -- 172.21.15.62:0/1717291854 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414400ac50 con 0x7f414c134b40 2024-04-19T01:49:20.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.133+0000 7f41427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:49:20.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.133+0000 7f41427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:49:20.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.133+0000 7f4143fff640 4 set_mon_vals no callback set 2024-04-19T01:49:20.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.133+0000 7f41427fc640 10 monclient: dump: 2024-04-19T01:49:20.138 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-04-19T01:49:20.138 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 6466e4b6-2520-485f-95b3-9401915fbbad 2024-04-19T01:49:20.138 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-04-19T01:41:38.497795+0000 2024-04-19T01:49:20.138 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-04-19T01:41:38.497795+0000 2024-04-19T01:49:20.138 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-04-19T01:49:20.138 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-04-19T01:49:20.138 INFO:teuthology.orchestra.run.smithi062.stderr:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-19T01:49:20.138 INFO:teuthology.orchestra.run.smithi062.stderr:1: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-19T01:49:20.138 INFO:teuthology.orchestra.run.smithi062.stderr:2: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-04-19T01:49:20.138 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:49:20.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.133+0000 7f4153b2f640 10 monclient: _renew_subs 2024-04-19T01:49:20.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.133+0000 7f4153b2f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:49:20.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.133+0000 7f4153b2f640 1 -- 172.21.15.62:0/1717291854 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f414c120cd0 con 0x7f414c134b40 2024-04-19T01:49:20.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.133+0000 7f4153b2f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:49:20.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.133+0000 7f4153b2f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:49:20.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.133+0000 7f4153b2f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:49:20.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.133+0000 7f4153b2f640 1 -- 172.21.15.62:0/1717291854 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f414c121090 con 0x7f414c134b40 2024-04-19T01:49:20.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.133+0000 7f41427fc640 1 -- 172.21.15.62:0/1717291854 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f414401a070 con 0x7f414c134b40 2024-04-19T01:49:20.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.133+0000 7f41427fc640 1 --2- 172.21.15.62:0/1717291854 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f41300364a0 0x7f4130038960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:49:20.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.133+0000 7f41427fc640 1 -- 172.21.15.62:0/1717291854 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(325..325 src has 115..325) ==== 4346+0+0 (secure 0 0 0) 0x7f41440489e0 con 0x7f414c134b40 2024-04-19T01:49:20.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.133+0000 7f41518cb640 1 --2- 172.21.15.62:0/1717291854 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f41300364a0 0x7f4130038960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:49:20.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.133+0000 7f41518cb640 10 monclient: get_auth_request con 0x7f41300364a0 auth_method 0 2024-04-19T01:49:20.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.137+0000 7f41518cb640 1 --2- 172.21.15.62:0/1717291854 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f41300364a0 0x7f4130038960 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f414c071740 tx=0x7f4148009060 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:49:20.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.137+0000 7f41427fc640 1 -- 172.21.15.62:0/1717291854 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f414400ce20 con 0x7f414c134b40 2024-04-19T01:49:20.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.137+0000 7f41427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:49:20.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.137+0000 7f41427fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:49:20.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.301+0000 7f4153b2f640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-04-19T01:49:20.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.301+0000 7f4153b2f640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-04-19T01:49:20.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.301+0000 7f4153b2f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.62:3300/0 2024-04-19T01:49:20.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.301+0000 7f4153b2f640 1 -- 172.21.15.62:0/1717291854 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f414c0692b0 con 0x7f414c134b40 2024-04-19T01:49:20.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.309+0000 7f41427fc640 1 -- 172.21.15.62:0/1717291854 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v5)=0 v5) ==== 123+0+0 (secure 0 0 0) 0x7f414401a360 con 0x7f414c134b40 2024-04-19T01:49:20.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.309+0000 7f41427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-04-19T01:49:20.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.309+0000 7f41427fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:49:20.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.309+0000 7f41427fc640 1 -- 172.21.15.62:0/1717291854 <== mon.0 v2:172.21.15.62:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f414404e080 con 0x7f414c134b40 2024-04-19T01:49:20.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.309+0000 7f41427fc640 10 monclient: handle_config config(1 keys) 2024-04-19T01:49:20.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.309+0000 7f41437fe640 4 set_mon_vals no callback set 2024-04-19T01:49:20.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.309+0000 7f41437fe640 10 set_mon_vals mon_health_to_clog = false 2024-04-19T01:49:20.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.313+0000 7f4153b2f640 1 -- 172.21.15.62:0/1717291854 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f41300364a0 msgr2=0x7f4130038960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:49:20.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.313+0000 7f4153b2f640 1 --2- 172.21.15.62:0/1717291854 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f41300364a0 0x7f4130038960 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f414c071740 tx=0x7f4148009060 comp rx=0 tx=0).stop 2024-04-19T01:49:20.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.313+0000 7f4153b2f640 10 monclient: shutdown 2024-04-19T01:49:20.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.313+0000 7f4153b2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:49:20.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.313+0000 7f4153b2f640 1 -- 172.21.15.62:0/1717291854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f414c134b40 msgr2=0x7f414c1cc1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:49:20.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.313+0000 7f4153b2f640 1 --2- 172.21.15.62:0/1717291854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f414c134b40 0x7f414c1cc1e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f414400c710 tx=0x7f414400cbd0 comp rx=0 tx=0).stop 2024-04-19T01:49:20.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.313+0000 7f4153b2f640 1 -- 172.21.15.62:0/1717291854 shutdown_connections 2024-04-19T01:49:20.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.313+0000 7f4153b2f640 1 --2- 172.21.15.62:0/1717291854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f414c134b40 0x7f414c1cc1e0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:49:20.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.313+0000 7f4153b2f640 1 --2- 172.21.15.62:0/1717291854 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f414c139710 0x7f414c134600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:49:20.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.313+0000 7f4153b2f640 1 --2- 172.21.15.62:0/1717291854 >> [v2:172.21.15.71:6824/2133533992,v1:172.21.15.71:6825/2133533992] conn(0x7f41300364a0 0x7f4130038960 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:49:20.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.313+0000 7f4153b2f640 1 --2- 172.21.15.62:0/1717291854 >> [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] conn(0x7f414c0a2ef0 0x7f414c071060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:49:20.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.313+0000 7f4153b2f640 1 -- 172.21.15.62:0/1717291854 >> 172.21.15.62:0/1717291854 conn(0x7f414c0a23d0 msgr2=0x7f414c135190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:49:20.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.313+0000 7f4153b2f640 1 -- 172.21.15.62:0/1717291854 shutdown_connections 2024-04-19T01:49:20.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-19T01:49:20.313+0000 7f4153b2f640 1 -- 172.21.15.62:0/1717291854 wait complete. 2024-04-19T01:49:20.325 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-19T01:49:20.325 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-19T01:49:20.325 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-19T01:49:20.326 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:49:20.403 INFO:tasks.ceph.osd.0:Stopped 2024-04-19T01:49:20.403 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-19T01:49:20.403 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:49:20.490 INFO:tasks.ceph.osd.1:Stopped 2024-04-19T01:49:20.491 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-19T01:49:20.491 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:49:26.270 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-04-19T01:49:26.593 INFO:tasks.ceph.osd.2:Stopped 2024-04-19T01:49:26.593 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-19T01:49:26.594 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:49:26.665 INFO:tasks.ceph.osd.3:Stopped 2024-04-19T01:49:26.665 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-19T01:49:26.665 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:49:26.763 INFO:tasks.ceph.osd.4:Stopped 2024-04-19T01:49:26.763 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-19T01:49:26.764 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:49:26.860 INFO:tasks.ceph.osd.5:Stopped 2024-04-19T01:49:26.860 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-19T01:49:26.860 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-19T01:49:26.860 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:49:26.890 INFO:tasks.ceph.mgr.x:Stopped 2024-04-19T01:49:26.890 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-19T01:49:26.890 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-19T01:49:26.890 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:49:26.914 INFO:tasks.ceph.mon.a:Stopped 2024-04-19T01:49:26.914 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-19T01:49:26.914 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:49:26.969 INFO:tasks.ceph.mon.c:Stopped 2024-04-19T01:49:26.969 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-19T01:49:26.969 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:49:26.983 INFO:tasks.ceph.mon.b:Stopped 2024-04-19T01:49:26.983 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-19T01:49:26.984 DEBUG:teuthology.orchestra.run.smithi062:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | head -n 1 2024-04-19T01:49:27.031 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi062.front.sepia.ceph.com 2024-04-19T01:49:27.032 DEBUG:teuthology.orchestra.run.smithi062:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-19T01:49:27.342 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi062.front.sepia.ceph.com 2024-04-19T01:49:27.342 DEBUG:teuthology.orchestra.run.smithi062:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-19T01:49:27.501 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi062.front.sepia.ceph.com 2024-04-19T01:49:27.501 DEBUG:teuthology.orchestra.run.smithi062:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-19T01:49:27.663 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi071.front.sepia.ceph.com 2024-04-19T01:49:27.663 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-19T01:49:27.808 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi071.front.sepia.ceph.com 2024-04-19T01:49:27.808 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-19T01:49:27.969 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi071.front.sepia.ceph.com 2024-04-19T01:49:27.969 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-19T01:49:28.137 INFO:tasks.ceph:Archiving mon data... 2024-04-19T01:49:28.138 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663309/data/mon.a.tgz 2024-04-19T01:49:28.138 DEBUG:teuthology.orchestra.run.smithi062:> mktemp 2024-04-19T01:49:28.144 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/tmp.LbLHbKLIIB 2024-04-19T01:49:28.144 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.LbLHbKLIIB 2024-04-19T01:49:28.234 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0666 /tmp/tmp.LbLHbKLIIB 2024-04-19T01:49:28.301 DEBUG:teuthology.orchestra.remote:smithi062:/tmp/tmp.LbLHbKLIIB is 172KB 2024-04-19T01:49:28.318 DEBUG:teuthology.orchestra.run.smithi062:> rm -fr /tmp/tmp.LbLHbKLIIB 2024-04-19T01:49:28.323 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663309/data/mon.c.tgz 2024-04-19T01:49:28.323 DEBUG:teuthology.orchestra.run.smithi062:> mktemp 2024-04-19T01:49:28.372 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/tmp.kUBDL36VJU 2024-04-19T01:49:28.372 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.kUBDL36VJU 2024-04-19T01:49:28.462 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0666 /tmp/tmp.kUBDL36VJU 2024-04-19T01:49:28.523 DEBUG:teuthology.orchestra.remote:smithi062:/tmp/tmp.kUBDL36VJU is 167KB 2024-04-19T01:49:28.544 DEBUG:teuthology.orchestra.run.smithi062:> rm -fr /tmp/tmp.kUBDL36VJU 2024-04-19T01:49:28.549 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663309/data/mon.b.tgz 2024-04-19T01:49:28.549 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2024-04-19T01:49:28.555 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.9RNBZn8BhD 2024-04-19T01:49:28.555 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.9RNBZn8BhD 2024-04-19T01:49:28.645 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.9RNBZn8BhD 2024-04-19T01:49:28.708 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.9RNBZn8BhD is 171KB 2024-04-19T01:49:28.726 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.9RNBZn8BhD 2024-04-19T01:49:28.731 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-19T01:49:28.731 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-19T01:49:28.734 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-19T01:49:28.796 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-19T01:49:28.831 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-19T01:49:28.831 INFO:tasks.ceph:Archiving crash dumps... 2024-04-19T01:49:28.833 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663309/remote/smithi062/crash 2024-04-19T01:49:28.834 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-19T01:49:28.853 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663309/remote/smithi071/crash 2024-04-19T01:49:28.854 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-19T01:49:28.885 INFO:tasks.ceph:Compressing logs... 2024-04-19T01:49:28.885 DEBUG:teuthology.orchestra.run.smithi062:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-19T01:49:28.901 DEBUG:teuthology.orchestra.run.smithi071:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-19T01:49:28.920 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27627.log 2024-04-19T01:49:28.921 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28419.log 2024-04-19T01:49:28.921 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.27627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27616.log 2024-04-19T01:49:28.922 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.28419.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27627.log.gz 2024-04-19T01:49:28.922 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27457.log 2024-04-19T01:49:28.922 INFO:teuthology.orchestra.run.smithi062.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28419.log.gz 2024-04-19T01:49:28.922 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30321.log 2024-04-19T01:49:28.922 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.27616.log: /var/log/ceph/ceph-client.admin.27457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30035.log 2024-04-19T01:49:28.923 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.30321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30347.log 2024-04-19T01:49:28.924 INFO:teuthology.orchestra.run.smithi062.stderr: 88.3%/var/log/ceph/ceph-client.admin.30035.log: -- replaced with /var/log/ceph/ceph-client.admin.27457.log.gz 2024-04-19T01:49:28.924 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30191.log 2024-04-19T01:49:28.924 INFO:teuthology.orchestra.run.smithi062.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.30321.log.gz 2024-04-19T01:49:28.924 INFO:teuthology.orchestra.run.smithi062.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.27616.log.gz 2024-04-19T01:49:28.924 INFO:teuthology.orchestra.run.smithi062.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.30035.log.gz 2024-04-19T01:49:28.924 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.30347.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-19T01:49:28.925 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.30191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27935.log 2024-04-19T01:49:28.925 INFO:teuthology.orchestra.run.smithi062.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.30347.log.gz 2024-04-19T01:49:28.925 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28523.log 2024-04-19T01:49:28.926 INFO:teuthology.orchestra.run.smithi062.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30191.log.gz 2024-04-19T01:49:28.926 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.27935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27781.log 2024-04-19T01:49:28.926 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.28523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27780.log 2024-04-19T01:49:28.926 INFO:teuthology.orchestra.run.smithi062.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.27935.log.gz 2024-04-19T01:49:28.931 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.27781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29983.log 2024-04-19T01:49:28.931 INFO:teuthology.orchestra.run.smithi062.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.28523.log.gz 2024-04-19T01:49:28.931 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.27780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30399.log 2024-04-19T01:49:28.931 INFO:teuthology.orchestra.run.smithi062.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.27781.log.gz 2024-04-19T01:49:28.931 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.29983.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27780.log.gz 2024-04-19T01:49:28.931 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23816.log 2024-04-19T01:49:28.931 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28619.log 2024-04-19T01:49:28.931 INFO:teuthology.orchestra.run.smithi062.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29983.log.gz 2024-04-19T01:49:28.931 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.30399.log: /var/log/ceph/ceph-client.admin.23816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30165.log 2024-04-19T01:49:28.931 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.28619.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.30399.log.gz 2024-04-19T01:49:28.931 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28235.log 2024-04-19T01:49:28.932 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.30165.log: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.23816.log.gz 2024-04-19T01:49:28.932 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29378.log 2024-04-19T01:49:28.932 INFO:teuthology.orchestra.run.smithi062.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.28619.log.gz 2024-04-19T01:49:28.932 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.28235.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-19T01:49:28.932 INFO:teuthology.orchestra.run.smithi062.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.30165.log.gz 2024-04-19T01:49:28.932 INFO:teuthology.orchestra.run.smithi062.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28235.log.gz 2024-04-19T01:49:28.932 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.29378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28261.log 2024-04-19T01:49:28.932 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27848.log 2024-04-19T01:49:28.932 INFO:teuthology.orchestra.run.smithi062.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29378.log.gz 2024-04-19T01:49:28.932 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.28261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28645.log 2024-04-19T01:49:28.932 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.27848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28341.log 2024-04-19T01:49:28.935 INFO:teuthology.orchestra.run.smithi062.stderr: 88.6% 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28261.log.gz 2024-04-19T01:49:28.935 INFO:teuthology.orchestra.run.smithi062.stderr: -- replaced with /var/log/ceph/ceph-client.admin.27848.log.gz 2024-04-19T01:49:28.935 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.28645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28863.log 2024-04-19T01:49:28.935 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.28341.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-19T01:49:28.935 INFO:teuthology.orchestra.run.smithi062.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.28341.log.gz 2024-04-19T01:49:28.935 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.28863.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.28645.log.gz 2024-04-19T01:49:28.935 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30113.log 2024-04-19T01:49:28.935 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27507.log/var/log/ceph/ceph-mon.a.log: 2024-04-19T01:49:28.936 INFO:teuthology.orchestra.run.smithi062.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28863.log.gz 2024-04-19T01:49:28.936 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.30113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27769.log 2024-04-19T01:49:28.936 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.27507.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.30113.log.gz 2024-04-19T01:49:28.936 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29536.log 2024-04-19T01:49:28.936 INFO:teuthology.orchestra.run.smithi062.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27507.log.gz 2024-04-19T01:49:28.936 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.27769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30477.log 2024-04-19T01:49:28.936 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28367.log 2024-04-19T01:49:28.936 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.29536.log: /var/log/ceph/ceph-client.admin.30477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29878.log 2024-04-19T01:49:28.936 INFO:teuthology.orchestra.run.smithi062.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.27769.log.gz 2024-04-19T01:49:28.941 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.28367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30425.log 2024-04-19T01:49:28.941 INFO:teuthology.orchestra.run.smithi062.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.29536.log.gz 2024-04-19T01:49:28.941 INFO:teuthology.orchestra.run.smithi062.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.30477.log.gz 2024-04-19T01:49:28.941 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.29878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27630.log 2024-04-19T01:49:28.941 INFO:teuthology.orchestra.run.smithi062.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.28367.log.gz 2024-04-19T01:49:28.941 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.30425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27971.log 2024-04-19T01:49:28.941 INFO:teuthology.orchestra.run.smithi062.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.29878.log.gz 2024-04-19T01:49:28.941 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.27630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29825.log 2024-04-19T01:49:28.941 INFO:teuthology.orchestra.run.smithi062.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.30425.log.gz 2024-04-19T01:49:28.941 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.27971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30009.log 2024-04-19T01:49:28.941 INFO:teuthology.orchestra.run.smithi062.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27630.log.gz 2024-04-19T01:49:28.942 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.29825.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27971.log.gz 2024-04-19T01:49:28.942 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30453.log 2024-04-19T01:49:28.942 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28967.log 2024-04-19T01:49:28.942 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.30009.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29825.log.gz 2024-04-19T01:49:28.942 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.30453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29562.log 2024-04-19T01:49:28.942 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.28967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28445.log 2024-04-19T01:49:28.942 INFO:teuthology.orchestra.run.smithi062.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.30009.log.gz 2024-04-19T01:49:28.942 INFO:teuthology.orchestra.run.smithi062.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.30453.log.gz 2024-04-19T01:49:28.942 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.29562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28157.log 2024-04-19T01:49:28.942 INFO:teuthology.orchestra.run.smithi062.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28967.log.gz 2024-04-19T01:49:28.942 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.28445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29019.log 2024-04-19T01:49:28.942 INFO:teuthology.orchestra.run.smithi062.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29562.log.gz 2024-04-19T01:49:28.942 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.28157.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.28445.log.gz 2024-04-19T01:49:28.942 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27776.log 2024-04-19T01:49:28.945 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29404.log 2024-04-19T01:49:28.945 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.29019.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28157.log.gz 2024-04-19T01:49:28.946 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.27776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24025.log 2024-04-19T01:49:28.946 INFO:teuthology.orchestra.run.smithi062.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.29019.log.gz 2024-04-19T01:49:28.946 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.29404.log: 89.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29074.log 2024-04-19T01:49:28.946 INFO:teuthology.orchestra.run.smithi062.stderr: -- replaced with /var/log/ceph/ceph-client.admin.27776.log.gz 2024-04-19T01:49:28.946 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.24025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29589.log 2024-04-19T01:49:28.946 INFO:teuthology.orchestra.run.smithi062.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29404.log.gz 2024-04-19T01:49:28.946 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.29074.log: gzip -5 --verbose 88.4% -- -- replaced with /var/log/ceph/ceph-client.admin.24025.log.gz /var/log/ceph/ceph-client.admin.29728.log 2024-04-19T01:49:28.950 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:49:28.950 INFO:teuthology.orchestra.run.smithi062.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.29074.log.gz 2024-04-19T01:49:28.950 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23934.log 2024-04-19T01:49:28.950 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.29589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28497.log 2024-04-19T01:49:28.950 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.29728.log: /var/log/ceph/ceph-client.admin.23934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27623.log 2024-04-19T01:49:28.950 INFO:teuthology.orchestra.run.smithi062.stderr: 89.4%/var/log/ceph/ceph-client.admin.28497.log: -- replaced with /var/log/ceph/ceph-client.admin.29728.log.gz 90.7% -- replaced with /var/log/ceph/ceph-client.admin.29589.log.gz 2024-04-19T01:49:28.950 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30269.log 2024-04-19T01:49:28.950 INFO:teuthology.orchestra.run.smithi062.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.23934.log.gz 2024-04-19T01:49:28.950 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:49:28.951 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.27623.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.28497.log.gz 2024-04-19T01:49:28.951 INFO:teuthology.orchestra.run.smithi062.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27623.log.gz 2024-04-19T01:49:28.991 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-19T01:49:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.30269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29322.log 2024-04-19T01:49:28.992 INFO:teuthology.orchestra.run.smithi062.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.30269.log.gz 2024-04-19T01:49:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30217.log 2024-04-19T01:49:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27765.log 2024-04-19T01:49:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.29322.log: /var/log/ceph/ceph-client.admin.30217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29456.log 2024-04-19T01:49:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.27765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28393.log 2024-04-19T01:49:28.992 INFO:teuthology.orchestra.run.smithi062.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29322.log.gz 2024-04-19T01:49:28.992 INFO:teuthology.orchestra.run.smithi062.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.30217.log.gz 2024-04-19T01:49:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.29456.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27765.log.gz 2024-04-19T01:49:28.992 INFO:teuthology.orchestra.run.smithi062.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.29456.log.gz 2024-04-19T01:49:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30061.log 2024-04-19T01:49:28.992 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.28393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29851.log 2024-04-19T01:49:28.993 INFO:teuthology.orchestra.run.smithi062.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28393.log.gz 2024-04-19T01:49:28.993 INFO:teuthology.orchestra.run.smithi062.stderr:gzip/var/log/ceph/ceph-client.admin.30061.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.27585.log 2024-04-19T01:49:28.993 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.29851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29643.log 2024-04-19T01:49:28.993 INFO:teuthology.orchestra.run.smithi062.stderr: 88.4%/var/log/ceph/ceph-client.admin.27585.log: -- replaced with /var/log/ceph/ceph-client.admin.30061.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28003.log 2024-04-19T01:49:28.993 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:49:28.993 INFO:teuthology.orchestra.run.smithi062.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29851.log.gz 2024-04-19T01:49:28.993 INFO:teuthology.orchestra.run.smithi062.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.27585.log.gz 2024-04-19T01:49:28.993 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.29643.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29643.log.gz 2024-04-19T01:49:28.993 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28315.log 2024-04-19T01:49:28.993 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.28003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28674.log 2024-04-19T01:49:28.993 INFO:teuthology.orchestra.run.smithi062.stderr: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-19T01:49:28.993 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.28315.log: -- replaced with /var/log/ceph/ceph-client.admin.28003.log.gz 2024-04-19T01:49:28.993 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.28674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28287.log 2024-04-19T01:49:28.993 INFO:teuthology.orchestra.run.smithi062.stderr: 89.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28731.log 2024-04-19T01:49:28.993 INFO:teuthology.orchestra.run.smithi062.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28315.log.gz 2024-04-19T01:49:28.994 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.28287.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.28674.log.gz 2024-04-19T01:49:28.994 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29430.log 2024-04-19T01:49:28.994 INFO:teuthology.orchestra.run.smithi062.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28287.log.gz 2024-04-19T01:49:28.994 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.28731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28079.log 2024-04-19T01:49:28.994 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.29430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27533.log 2024-04-19T01:49:28.994 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.28079.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.28731.log.gz 2024-04-19T01:49:28.994 INFO:teuthology.orchestra.run.smithi062.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29430.log.gz 2024-04-19T01:49:28.994 INFO:teuthology.orchestra.run.smithi062.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28079.log.gz 2024-04-19T01:49:28.994 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29931.log 2024-04-19T01:49:28.994 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29296.log 2024-04-19T01:49:28.994 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.27533.log: /var/log/ceph/ceph-client.admin.29931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29045.log 2024-04-19T01:49:28.994 INFO:teuthology.orchestra.run.smithi062.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.27533.log.gz 2024-04-19T01:49:28.994 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.29296.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.29931.log.gz 2024-04-19T01:49:28.994 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28131.log 2024-04-19T01:49:28.995 INFO:teuthology.orchestra.run.smithi062.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.29296.log.gz 2024-04-19T01:49:28.995 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.29045.log: 91.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-19T01:49:28.995 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28941.log 2024-04-19T01:49:28.995 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.28131.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-19T01:49:28.995 INFO:teuthology.orchestra.run.smithi062.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.29045.log.gz 2024-04-19T01:49:28.995 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.28941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27631.log 2024-04-19T01:49:28.995 INFO:teuthology.orchestra.run.smithi062.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28131.log.gz 2024-04-19T01:49:28.995 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29510.log 2024-04-19T01:49:28.995 INFO:teuthology.orchestra.run.smithi062.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.28941.log.gz 2024-04-19T01:49:28.995 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.27631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28703.log 2024-04-19T01:49:28.995 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28915.log 2024-04-19T01:49:28.995 INFO:teuthology.orchestra.run.smithi062.stderr: 88.0%/var/log/ceph/ceph-client.admin.29510.log: -- replaced with /var/log/ceph/ceph-client.admin.27631.log.gz 2024-04-19T01:49:28.995 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.28703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27930.log 2024-04-19T01:49:28.995 INFO:teuthology.orchestra.run.smithi062.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.29510.log.gz 2024-04-19T01:49:28.995 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.28915.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.28703.log.gz 2024-04-19T01:49:28.995 INFO:teuthology.orchestra.run.smithi062.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.28915.log.gz 2024-04-19T01:49:28.996 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-19T01:49:28.996 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.27930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30243.log 2024-04-19T01:49:28.996 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27624.log 2024-04-19T01:49:28.996 INFO:teuthology.orchestra.run.smithi062.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.27930.log.gz 2024-04-19T01:49:28.996 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.30243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29100.log 2024-04-19T01:49:28.996 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.27624.log: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-19T01:49:28.996 INFO:teuthology.orchestra.run.smithi062.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.30243.log.gz 2024-04-19T01:49:28.996 INFO:teuthology.orchestra.run.smithi062.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27624.log.gz 2024-04-19T01:49:28.996 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28835.log 2024-04-19T01:49:28.996 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28471.log 2024-04-19T01:49:28.996 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.29100.log: /var/log/ceph/ceph-client.admin.28835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29904.log 2024-04-19T01:49:28.996 INFO:teuthology.orchestra.run.smithi062.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.29100.log.gz 2024-04-19T01:49:28.996 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.28471.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28835.log.gz 2024-04-19T01:49:28.996 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28809.log 2024-04-19T01:49:28.996 INFO:teuthology.orchestra.run.smithi062.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.28471.log.gz 2024-04-19T01:49:28.997 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.29904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30139.log 2024-04-19T01:49:28.997 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.28809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30373.log 2024-04-19T01:49:28.997 INFO:teuthology.orchestra.run.smithi062.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.29904.log.gz/var/log/ceph/ceph-client.admin.30139.log: 2024-04-19T01:49:28.997 INFO:teuthology.orchestra.run.smithi062.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.28809.log.gz 2024-04-19T01:49:28.997 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28554.log 2024-04-19T01:49:28.997 INFO:teuthology.orchestra.run.smithi062.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30139.log.gz 2024-04-19T01:49:28.997 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.30373.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.30373.log.gz 2024-04-19T01:49:28.997 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29242.log 2024-04-19T01:49:28.997 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.28554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27559.log 2024-04-19T01:49:28.997 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.29242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28993.log 2024-04-19T01:49:28.997 INFO:teuthology.orchestra.run.smithi062.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28554.log.gz 2024-04-19T01:49:28.997 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.27559.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29242.log.gz 2024-04-19T01:49:28.997 INFO:teuthology.orchestra.run.smithi062.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27559.log.gz 2024-04-19T01:49:28.997 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27426.log 2024-04-19T01:49:28.997 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.28993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27923.log 2024-04-19T01:49:28.997 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.27426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29348.log 2024-04-19T01:49:28.998 INFO:teuthology.orchestra.run.smithi062.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.28993.log.gz 2024-04-19T01:49:28.998 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.27923.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.27426.log.gz 2024-04-19T01:49:28.998 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20230.log 2024-04-19T01:49:28.998 INFO:teuthology.orchestra.run.smithi062.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27923.log.gz 2024-04-19T01:49:28.998 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.29348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28889.log 2024-04-19T01:49:28.998 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph.tmp-client.admin.20230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29957.log 2024-04-19T01:49:28.998 INFO:teuthology.orchestra.run.smithi062.stderr: 0.0% 91.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20230.log.gz 2024-04-19T01:49:28.998 INFO:teuthology.orchestra.run.smithi062.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29348.log.gz 2024-04-19T01:49:28.998 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.28889.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.28889.log.gz 2024-04-19T01:49:28.998 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30295.log 2024-04-19T01:49:28.998 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.29957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29797.log 2024-04-19T01:49:28.998 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.30295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28757.log 2024-04-19T01:49:28.998 INFO:teuthology.orchestra.run.smithi062.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.29957.log.gz 2024-04-19T01:49:28.998 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.29797.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30295.log.gz 2024-04-19T01:49:28.999 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29186.log 2024-04-19T01:49:28.999 INFO:teuthology.orchestra.run.smithi062.stderr: 89.6%/var/log/ceph/ceph-client.admin.28757.log: -- replaced with /var/log/ceph/ceph-client.admin.29797.log.gz 2024-04-19T01:49:28.999 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27931.log 2024-04-19T01:49:28.999 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.29186.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.28757.log.gz 2024-04-19T01:49:28.999 INFO:teuthology.orchestra.run.smithi062.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29186.log.gz 2024-04-19T01:49:28.999 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28105.log 2024-04-19T01:49:28.999 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.27931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29484.log 2024-04-19T01:49:28.999 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.28105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29214.log 2024-04-19T01:49:28.999 INFO:teuthology.orchestra.run.smithi062.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27931.log.gz 2024-04-19T01:49:28.999 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.29484.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.28105.log.gz 2024-04-19T01:49:28.999 INFO:teuthology.orchestra.run.smithi062.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.29484.log.gz 2024-04-19T01:49:28.999 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28209.log 2024-04-19T01:49:28.999 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.29214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28183.log 2024-04-19T01:49:28.999 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.28209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30087.log 2024-04-19T01:49:28.999 INFO:teuthology.orchestra.run.smithi062.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29214.log.gz 2024-04-19T01:49:29.000 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.28183.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28209.log.gz 2024-04-19T01:49:29.000 INFO:teuthology.orchestra.run.smithi062.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.28183.log.gz 2024-04-19T01:49:29.000 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29270.log 2024-04-19T01:49:29.000 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.30087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28783.log 2024-04-19T01:49:29.000 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.29270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29617.log 2024-04-19T01:49:29.000 INFO:teuthology.orchestra.run.smithi062.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.30087.log.gz 2024-04-19T01:49:29.000 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.28783.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29270.log.gz 2024-04-19T01:49:29.000 INFO:teuthology.orchestra.run.smithi062.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.28783.log.gz 2024-04-19T01:49:29.000 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.29617.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.29617.log.gz 2024-04-19T01:49:29.050 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31129.log 2024-04-19T01:49:29.050 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27909.log 2024-04-19T01:49:29.050 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30957.log 2024-04-19T01:49:29.050 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.27909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28419.log 2024-04-19T01:49:29.050 INFO:teuthology.orchestra.run.smithi071.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.31129.log.gz 2024-04-19T01:49:29.050 INFO:teuthology.orchestra.run.smithi071.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27909.log.gz 2024-04-19T01:49:29.050 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30607.log 2024-04-19T01:49:29.051 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30957.log: /var/log/ceph/ceph-client.admin.28419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31305.log 2024-04-19T01:49:29.051 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29346.log 2024-04-19T01:49:29.051 INFO:teuthology.orchestra.run.smithi071.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.28419.log.gz 2024-04-19T01:49:29.051 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28090.log 2024-04-19T01:49:29.051 INFO:teuthology.orchestra.run.smithi071.stderr:gzip/var/log/ceph/ceph-client.admin.29346.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30431.log 2024-04-19T01:49:29.051 INFO:teuthology.orchestra.run.smithi071.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.31305.log.gz 2024-04-19T01:49:29.051 INFO:teuthology.orchestra.run.smithi071.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.30607.log.gz 2024-04-19T01:49:29.051 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28090.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.30957.log.gz 2024-04-19T01:49:29.051 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29693.log 2024-04-19T01:49:29.051 INFO:teuthology.orchestra.run.smithi071.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.29346.log.gz 2024-04-19T01:49:29.051 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27600.log 2024-04-19T01:49:29.051 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29607.log 2024-04-19T01:49:29.051 INFO:teuthology.orchestra.run.smithi071.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.28090.log.gz 2024-04-19T01:49:29.051 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.27600.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.30431.log.gz 2024-04-19T01:49:29.051 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27636.log 2024-04-19T01:49:29.052 INFO:teuthology.orchestra.run.smithi071.stderr:gzip/var/log/ceph/ceph-client.admin.29607.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.27818.log 2024-04-19T01:49:29.052 INFO:teuthology.orchestra.run.smithi071.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.27600.log.gz 2024-04-19T01:49:29.052 INFO:teuthology.orchestra.run.smithi071.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.29693.log.gz 2024-04-19T01:49:29.052 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.27636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29572.log 2024-04-19T01:49:29.052 INFO:teuthology.orchestra.run.smithi071.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29607.log.gz 2024-04-19T01:49:29.052 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.27818.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.27636.log.gz 2024-04-19T01:49:29.052 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28055.log 2024-04-19T01:49:29.052 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29398.log 2024-04-19T01:49:29.052 INFO:teuthology.orchestra.run.smithi071.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.27818.log.gz 2024-04-19T01:49:29.052 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29399.log 2024-04-19T01:49:29.052 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29834.log 2024-04-19T01:49:29.052 INFO:teuthology.orchestra.run.smithi071.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.29572.log.gz 2024-04-19T01:49:29.052 INFO:teuthology.orchestra.run.smithi071.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28055.log.gz 2024-04-19T01:49:29.052 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28602.log 2024-04-19T01:49:29.053 INFO:teuthology.orchestra.run.smithi071.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29398.log.gz 2024-04-19T01:49:29.053 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29834.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.29399.log.gz 2024-04-19T01:49:29.053 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31564.log 2024-04-19T01:49:29.053 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30956.log 2024-04-19T01:49:29.053 INFO:teuthology.orchestra.run.smithi071.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.29834.log.gz 2024-04-19T01:49:29.053 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28511.log 2024-04-19T01:49:29.053 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29746.log 2024-04-19T01:49:29.053 INFO:teuthology.orchestra.run.smithi071.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.28602.log.gz 2024-04-19T01:49:29.053 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28511.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30956.log.gz 2024-04-19T01:49:29.053 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28639.log 2024-04-19T01:49:29.053 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29224.log 2024-04-19T01:49:29.053 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29746.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31564.log.gz 2024-04-19T01:49:29.053 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27781.log 2024-04-19T01:49:29.053 INFO:teuthology.orchestra.run.smithi071.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.28511.log.gz 2024-04-19T01:49:29.053 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31008.log 2024-04-19T01:49:29.053 INFO:teuthology.orchestra.run.smithi071.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28639.log.gz 2024-04-19T01:49:29.054 INFO:teuthology.orchestra.run.smithi071.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29746.log.gz 2024-04-19T01:49:29.054 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.27781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30783.log 2024-04-19T01:49:29.054 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30748.log 2024-04-19T01:49:29.054 INFO:teuthology.orchestra.run.smithi071.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.29224.log.gz 2024-04-19T01:49:29.054 INFO:teuthology.orchestra.run.smithi071.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.27781.log.gz 2024-04-19T01:49:29.054 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30432.log 2024-04-19T01:49:29.054 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30094.log 2024-04-19T01:49:29.054 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30432.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.31008.log.gz 2024-04-19T01:49:29.054 INFO:teuthology.orchestra.run.smithi071.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.30748.log.gz 2024-04-19T01:49:29.054 INFO:teuthology.orchestra.run.smithi071.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.30783.log.gz 2024-04-19T01:49:29.054 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28822.log 2024-04-19T01:49:29.054 INFO:teuthology.orchestra.run.smithi071.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.30432.log.gz 2024-04-19T01:49:29.054 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28908.log 2024-04-19T01:49:29.054 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28821.log 2024-04-19T01:49:29.054 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28603.log 2024-04-19T01:49:29.055 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28821.log: gzip 89.1% 89.8% -- replaced with /var/log/ceph/ceph-client.admin.28908.log.gz -5 -- replaced with /var/log/ceph/ceph-client.admin.30094.log.gz 2024-04-19T01:49:29.055 INFO:teuthology.orchestra.run.smithi071.stderr: --verbose 2024-04-19T01:49:29.055 INFO:teuthology.orchestra.run.smithi071.stderr: -- /var/log/ceph/ceph-client.admin.29956.log 2024-04-19T01:49:29.055 INFO:teuthology.orchestra.run.smithi071.stderr:gzip 89.8% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.28821.log.gz -- 2024-04-19T01:49:29.055 INFO:teuthology.orchestra.run.smithi071.stderr: 90.8% /var/log/ceph/ceph-client.admin.27728.log 2024-04-19T01:49:29.055 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28822.log.gz 2024-04-19T01:49:29.055 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28603.log: /var/log/ceph/ceph-client.admin.29956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30130.log 2024-04-19T01:49:29.055 INFO:teuthology.orchestra.run.smithi071.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.28603.log.gz 2024-04-19T01:49:29.055 INFO:teuthology.orchestra.run.smithi071.stderr:gzip/var/log/ceph/ceph-client.admin.27728.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.31009.log 2024-04-19T01:49:29.055 INFO:teuthology.orchestra.run.smithi071.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.29956.log.gz 2024-04-19T01:49:29.055 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27635.log 2024-04-19T01:49:29.055 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29171.log 2024-04-19T01:49:29.055 INFO:teuthology.orchestra.run.smithi071.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.27728.log.gz 2024-04-19T01:49:29.055 INFO:teuthology.orchestra.run.smithi071.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.30130.log.gz 2024-04-19T01:49:29.055 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.27635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31042.log 2024-04-19T01:49:29.056 INFO:teuthology.orchestra.run.smithi071.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.31009.log.gz 2024-04-19T01:49:29.056 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30310.log 2024-04-19T01:49:29.056 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30521.log 2024-04-19T01:49:29.056 INFO:teuthology.orchestra.run.smithi071.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.27635.log.gz 2024-04-19T01:49:29.056 INFO:teuthology.orchestra.run.smithi071.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29171.log.gz 2024-04-19T01:49:29.056 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29833.log 2024-04-19T01:49:29.056 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29260.log 2024-04-19T01:49:29.056 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29833.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.31042.log.gz 2024-04-19T01:49:29.056 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31216.log 2024-04-19T01:49:29.056 INFO:teuthology.orchestra.run.smithi071.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.30521.log.gz 2024-04-19T01:49:29.056 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30868.log 2024-04-19T01:49:29.056 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29260.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29833.log.gz 2024-04-19T01:49:29.056 INFO:teuthology.orchestra.run.smithi071.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.30310.log.gz 2024-04-19T01:49:29.056 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27509.log 2024-04-19T01:49:29.057 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.30868.log: /var/log/ceph/ceph-client.admin.29432.log 2024-04-19T01:49:29.057 INFO:teuthology.orchestra.run.smithi071.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.29260.log.gz 2024-04-19T01:49:29.057 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.27509.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.31216.log.gz 2024-04-19T01:49:29.057 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29051.log 2024-04-19T01:49:29.057 INFO:teuthology.orchestra.run.smithi071.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.27509.log.gz 2024-04-19T01:49:29.057 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28549.log 2024-04-19T01:49:29.057 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29311.log 2024-04-19T01:49:29.057 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28549.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.30868.log.gz 2024-04-19T01:49:29.057 INFO:teuthology.orchestra.run.smithi071.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29051.log.gz 2024-04-19T01:49:29.057 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28907.log 2024-04-19T01:49:29.057 INFO:teuthology.orchestra.run.smithi071.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29432.log.gz 2024-04-19T01:49:29.057 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31653.log 2024-04-19T01:49:29.057 INFO:teuthology.orchestra.run.smithi071.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.28549.log.gz 2024-04-19T01:49:29.057 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27508.log 2024-04-19T01:49:29.057 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30609.log 2024-04-19T01:49:29.058 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31653.log: /var/log/ceph/ceph-client.admin.27508.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.29311.log.gz 2024-04-19T01:49:29.058 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30095.log 2024-04-19T01:49:29.058 INFO:teuthology.orchestra.run.smithi071.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.28907.log.gz 2024-04-19T01:49:29.058 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31217.log 2024-04-19T01:49:29.058 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30609.log: /var/log/ceph/ceph-client.admin.30095.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.27508.log.gz 2024-04-19T01:49:29.058 INFO:teuthology.orchestra.run.smithi071.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.31653.log.gz 2024-04-19T01:49:29.058 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29606.log 2024-04-19T01:49:29.058 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30519.log 2024-04-19T01:49:29.058 INFO:teuthology.orchestra.run.smithi071.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.30095.log.gz 2024-04-19T01:49:29.058 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29606.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.30609.log.gz 2024-04-19T01:49:29.058 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31390.log 2024-04-19T01:49:29.058 INFO:teuthology.orchestra.run.smithi071.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31217.log.gz 2024-04-19T01:49:29.058 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31565.log 2024-04-19T01:49:29.058 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31652.log 2024-04-19T01:49:29.058 INFO:teuthology.orchestra.run.smithi071.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.29606.log.gz 2024-04-19T01:49:29.059 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28091.log 2024-04-19T01:49:29.059 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31652.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.30519.log.gz 2024-04-19T01:49:29.059 INFO:teuthology.orchestra.run.smithi071.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.31565.log.gz 2024-04-19T01:49:29.059 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31478.log 2024-04-19T01:49:29.059 INFO:teuthology.orchestra.run.smithi071.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.31652.log.gz 2024-04-19T01:49:29.059 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29312.log 2024-04-19T01:49:29.059 INFO:teuthology.orchestra.run.smithi071.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.31390.log.gz 2024-04-19T01:49:29.059 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28091.log: /var/log/ceph/ceph-client.admin.31478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31304.log 2024-04-19T01:49:29.059 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31566.log 2024-04-19T01:49:29.059 INFO:teuthology.orchestra.run.smithi071.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28091.log.gz 2024-04-19T01:49:29.059 INFO:teuthology.orchestra.run.smithi071.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.31478.log.gz 2024-04-19T01:49:29.059 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28274.log 2024-04-19T01:49:29.059 INFO:teuthology.orchestra.run.smithi071.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29312.log.gz 2024-04-19T01:49:29.059 INFO:teuthology.orchestra.run.smithi071.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.31304.log.gz 2024-04-19T01:49:29.059 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29050.log 2024-04-19T01:49:29.060 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29955.log 2024-04-19T01:49:29.060 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28997.log 2024-04-19T01:49:29.060 INFO:teuthology.orchestra.run.smithi071.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.31566.log.gz 2024-04-19T01:49:29.060 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28001.log 2024-04-19T01:49:29.060 INFO:teuthology.orchestra.run.smithi071.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.29955.log.gz 2024-04-19T01:49:29.060 INFO:teuthology.orchestra.run.smithi071.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.28274.log.gz 2024-04-19T01:49:29.060 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29659.log 2024-04-19T01:49:29.060 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28997.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.29050.log.gz 2024-04-19T01:49:29.060 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28694.log 2024-04-19T01:49:29.060 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28001.log: /var/log/ceph/ceph-client.admin.29659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30696.log 2024-04-19T01:49:29.060 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28694.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.28997.log.gz 2024-04-19T01:49:29.060 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28693.log 2024-04-19T01:49:29.060 INFO:teuthology.orchestra.run.smithi071.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28694.log.gz 2024-04-19T01:49:29.060 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30694.log 2024-04-19T01:49:29.061 INFO:teuthology.orchestra.run.smithi071.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.29659.log.gz 2024-04-19T01:49:29.061 INFO:teuthology.orchestra.run.smithi071.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.28001.log.gz 2024-04-19T01:49:29.061 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30696.log: /var/log/ceph/ceph-client.admin.28693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30870.log 2024-04-19T01:49:29.061 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29172.log 2024-04-19T01:49:29.061 INFO:teuthology.orchestra.run.smithi071.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.30696.log.gz 2024-04-19T01:49:29.061 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28964.log 2024-04-19T01:49:29.061 INFO:teuthology.orchestra.run.smithi071.stderr: 89.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.30694.log.gz 2024-04-19T01:49:29.061 INFO:teuthology.orchestra.run.smithi071.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.29085.log 2024-04-19T01:49:29.061 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29172.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.28693.log.gz 2024-04-19T01:49:29.061 INFO:teuthology.orchestra.run.smithi071.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.30870.log.gz 2024-04-19T01:49:29.061 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28873.log 2024-04-19T01:49:29.061 INFO:teuthology.orchestra.run.smithi071.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29172.log.gz 2024-04-19T01:49:29.061 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5/var/log/ceph/ceph-client.admin.29085.log: --verbose -- /var/log/ceph/ceph-client.admin.29138.log 2024-04-19T01:49:29.061 INFO:teuthology.orchestra.run.smithi071.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28964.log.gz 2024-04-19T01:49:29.061 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31443.log 2024-04-19T01:49:29.061 INFO:teuthology.orchestra.run.smithi071.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29085.log.gz 2024-04-19T01:49:29.062 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29138.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-19T01:49:29.062 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30922.log 2024-04-19T01:49:29.062 INFO:teuthology.orchestra.run.smithi071.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.28873.log.gz 2024-04-19T01:49:29.062 INFO:teuthology.orchestra.run.smithi071.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29138.log.gz 2024-04-19T01:49:29.062 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30433.log 2024-04-19T01:49:29.062 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28145.log 2024-04-19T01:49:29.062 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30433.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.30922.log.gz 2024-04-19T01:49:29.062 INFO:teuthology.orchestra.run.smithi071.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.31443.log.gz 2024-04-19T01:49:29.062 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-19T01:49:29.062 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30346.log 2024-04-19T01:49:29.062 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28145.log: /var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28273.log 2024-04-19T01:49:29.062 INFO:teuthology.orchestra.run.smithi071.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.30433.log.gz 2024-04-19T01:49:29.062 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30128.log 2024-04-19T01:49:29.062 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28273.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.28145.log.gz 2024-04-19T01:49:29.062 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31705.log 2024-04-19T01:49:29.063 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30041.log 2024-04-19T01:49:29.063 INFO:teuthology.orchestra.run.smithi071.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.28273.log.gz 2024-04-19T01:49:29.063 INFO:teuthology.orchestra.run.smithi071.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.30346.log.gz 2024-04-19T01:49:29.063 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23840.log 2024-04-19T01:49:29.063 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31740.log 2024-04-19T01:49:29.063 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30041.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31705.log.gz 2024-04-19T01:49:29.063 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.23840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30608.log 2024-04-19T01:49:29.063 INFO:teuthology.orchestra.run.smithi071.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.30128.log.gz 2024-04-19T01:49:29.063 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31391.log 2024-04-19T01:49:29.063 INFO:teuthology.orchestra.run.smithi071.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.23840.log.gz 2024-04-19T01:49:29.063 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29608.log 2024-04-19T01:49:29.063 INFO:teuthology.orchestra.run.smithi071.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.30041.log.gz 2024-04-19T01:49:29.063 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27910.log 2024-04-19T01:49:29.063 INFO:teuthology.orchestra.run.smithi071.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.30608.log.gz 2024-04-19T01:49:29.064 INFO:teuthology.orchestra.run.smithi071.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.31740.log.gz 2024-04-19T01:49:29.064 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31792.log 2024-04-19T01:49:29.064 INFO:teuthology.orchestra.run.smithi071.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.31391.log.gz 2024-04-19T01:49:29.064 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30217.log 2024-04-19T01:49:29.064 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.27910.log: /var/log/ceph/ceph-client.admin.31792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30955.log 2024-04-19T01:49:29.064 INFO:teuthology.orchestra.run.smithi071.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.29608.log.gz 2024-04-19T01:49:29.064 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27546.log 2024-04-19T01:49:29.064 INFO:teuthology.orchestra.run.smithi071.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.27910.log.gz 2024-04-19T01:49:29.064 INFO:teuthology.orchestra.run.smithi071.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.31792.log.gz 2024-04-19T01:49:29.064 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29869.log 2024-04-19T01:49:29.064 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.27546.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.30217.log.gz 2024-04-19T01:49:29.064 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29954.log 2024-04-19T01:49:29.064 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31096.log 2024-04-19T01:49:29.064 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29869.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.30955.log.gz 2024-04-19T01:49:29.064 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28327.log 2024-04-19T01:49:29.065 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31096.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.27546.log.gz 2024-04-19T01:49:29.065 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30834.log 2024-04-19T01:49:29.065 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28327.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.31096.log.gz 2024-04-19T01:49:29.065 INFO:teuthology.orchestra.run.smithi071.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.29869.log.gz 2024-04-19T01:49:29.065 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28512.log 2024-04-19T01:49:29.065 INFO:teuthology.orchestra.run.smithi071.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.29954.log.gz 2024-04-19T01:49:29.065 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31095.log 2024-04-19T01:49:29.065 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30834.log: /var/log/ceph/ceph-client.admin.28512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31044.log 2024-04-19T01:49:29.065 INFO:teuthology.orchestra.run.smithi071.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.28327.log.gz 2024-04-19T01:49:29.065 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31095.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28512.log.gz 2024-04-19T01:49:29.065 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27908.log 2024-04-19T01:49:29.065 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23912.log 2024-04-19T01:49:29.065 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31044.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.30834.log.gz 2024-04-19T01:49:29.065 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.27908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31477.log 2024-04-19T01:49:29.065 INFO:teuthology.orchestra.run.smithi071.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.31095.log.gz 2024-04-19T01:49:29.066 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.23912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31738.log 2024-04-19T01:49:29.066 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30345.log 2024-04-19T01:49:29.066 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31477.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.23912.log.gz 2024-04-19T01:49:29.066 INFO:teuthology.orchestra.run.smithi071.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.31044.log.gz 2024-04-19T01:49:29.066 INFO:teuthology.orchestra.run.smithi071.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.27908.log.gz 2024-04-19T01:49:29.066 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31131.log 2024-04-19T01:49:29.066 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30181.log 2024-04-19T01:49:29.066 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30345.log: /var/log/ceph/ceph-client.admin.31131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29694.log 2024-04-19T01:49:29.066 INFO:teuthology.orchestra.run.smithi071.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.31477.log.gz 2024-04-19T01:49:29.066 INFO:teuthology.orchestra.run.smithi071.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30345.log.gz 2024-04-19T01:49:29.066 INFO:teuthology.orchestra.run.smithi071.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31738.log.gz 2024-04-19T01:49:29.066 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29434.log 2024-04-19T01:49:29.066 INFO:teuthology.orchestra.run.smithi071.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.31131.log.gz 2024-04-19T01:49:29.066 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27599.log 2024-04-19T01:49:29.066 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30216.log 2024-04-19T01:49:29.067 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29434.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.29694.log.gz 2024-04-19T01:49:29.067 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.27599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28181.log 2024-04-19T01:49:29.067 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30216.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.30181.log.gz 2024-04-19T01:49:29.067 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-19T01:49:29.067 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28181.log: 89.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28456.log 2024-04-19T01:49:29.067 INFO:teuthology.orchestra.run.smithi071.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30216.log.gz 2024-04-19T01:49:29.067 INFO:teuthology.orchestra.run.smithi071.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.29434.log.gz 2024-04-19T01:49:29.067 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.27599.log.gz 2024-04-19T01:49:29.067 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27691.log 2024-04-19T01:49:29.067 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30574.log 2024-04-19T01:49:29.067 INFO:teuthology.orchestra.run.smithi071.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.28181.log.gz 2024-04-19T01:49:29.067 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31739.log 2024-04-19T01:49:29.067 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.27691.log: /var/log/ceph/ceph-client.admin.30574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29781.log 2024-04-19T01:49:29.067 INFO:teuthology.orchestra.run.smithi071.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.28456.log.gz 2024-04-19T01:49:29.067 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31739.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.27691.log.gz 2024-04-19T01:49:29.068 INFO:teuthology.orchestra.run.smithi071.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.30574.log.gz 2024-04-19T01:49:29.068 INFO:teuthology.orchestra.run.smithi071.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.31739.log.gz 2024-04-19T01:49:29.068 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28729.log 2024-04-19T01:49:29.068 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29521.log 2024-04-19T01:49:29.068 INFO:teuthology.orchestra.run.smithi071.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.29781.log.gz 2024-04-19T01:49:29.068 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30043.log 2024-04-19T01:49:29.068 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28729.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-19T01:49:29.068 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29521.log: /var/log/ceph/ceph-client.admin.30043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30007.log 2024-04-19T01:49:29.068 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27872.log 2024-04-19T01:49:29.068 INFO:teuthology.orchestra.run.smithi071.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.29521.log.gz 2024-04-19T01:49:29.068 INFO:teuthology.orchestra.run.smithi071.stderr: 90.4% 89.4% -- replaced with /var/log/ceph/ceph-client.admin.28729.log.gz 2024-04-19T01:49:29.068 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30043.log.gz 2024-04-19T01:49:29.068 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30007.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.30007.log.gz 2024-04-19T01:49:29.068 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29485.log 2024-04-19T01:49:29.068 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.27872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29573.log 2024-04-19T01:49:29.068 INFO:teuthology.orchestra.run.smithi071.stderr: 87.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-19T01:49:29.069 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31130.log 2024-04-19T01:49:29.069 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28731.log 2024-04-19T01:49:29.069 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29573.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.27872.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30008.log 2024-04-19T01:49:29.069 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31130.log: 2024-04-19T01:49:29.069 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28731.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.29485.log.gz 2024-04-19T01:49:29.069 INFO:teuthology.orchestra.run.smithi071.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29573.log.gz 2024-04-19T01:49:29.069 INFO:teuthology.orchestra.run.smithi071.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.31130.log.gz 2024-04-19T01:49:29.069 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29921.log 2024-04-19T01:49:29.069 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28730.log 2024-04-19T01:49:29.069 INFO:teuthology.orchestra.run.smithi071.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.28731.log.gz 2024-04-19T01:49:29.069 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29921.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.30008.log.gz 2024-04-19T01:49:29.069 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29520.log 2024-04-19T01:49:29.069 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29780.log 2024-04-19T01:49:29.069 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28730.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29921.log.gz 2024-04-19T01:49:29.069 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28146.log 2024-04-19T01:49:29.070 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29780.log: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31826.log 2024-04-19T01:49:29.070 INFO:teuthology.orchestra.run.smithi071.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28730.log.gz 2024-04-19T01:49:29.070 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29520.log.gz 2024-04-19T01:49:29.070 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28365.log 2024-04-19T01:49:29.070 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27727.log 2024-04-19T01:49:29.070 INFO:teuthology.orchestra.run.smithi071.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28146.log.gz 2024-04-19T01:49:29.070 INFO:teuthology.orchestra.run.smithi071.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.29780.log.gz 2024-04-19T01:49:29.070 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31444.log 2024-04-19T01:49:29.070 INFO:teuthology.orchestra.run.smithi071.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.31826.log.gz 2024-04-19T01:49:29.070 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.27727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29347.log 2024-04-19T01:49:29.070 INFO:teuthology.orchestra.run.smithi071.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.28365.log.gz 2024-04-19T01:49:29.070 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27545.log 2024-04-19T01:49:29.070 INFO:teuthology.orchestra.run.smithi071.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.27727.log.gz 2024-04-19T01:49:29.070 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29137.log 2024-04-19T01:49:29.070 INFO:teuthology.orchestra.run.smithi071.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31444.log.gz 2024-04-19T01:49:29.071 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.27545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29782.log 2024-04-19T01:49:29.071 INFO:teuthology.orchestra.run.smithi071.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.29347.log.gz 2024-04-19T01:49:29.071 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31825.log 2024-04-19T01:49:29.071 INFO:teuthology.orchestra.run.smithi071.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27545.log.gz 2024-04-19T01:49:29.071 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30398.log 2024-04-19T01:49:29.071 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29782.log: /var/log/ceph/ceph-client.admin.31825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31303.log 2024-04-19T01:49:29.071 INFO:teuthology.orchestra.run.smithi071.stderr: 89.5%/var/log/ceph/ceph-client.admin.30398.log: -- replaced with /var/log/ceph/ceph-client.admin.29137.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28963.log 2024-04-19T01:49:29.071 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:49:29.071 INFO:teuthology.orchestra.run.smithi071.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.29782.log.gz 2024-04-19T01:49:29.071 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31303.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.30398.log.gz 2024-04-19T01:49:29.071 INFO:teuthology.orchestra.run.smithi071.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.31825.log.gz 2024-04-19T01:49:29.071 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30921.log 2024-04-19T01:49:29.071 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29258.log 2024-04-19T01:49:29.071 INFO:teuthology.orchestra.run.smithi071.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.31303.log.gz 2024-04-19T01:49:29.071 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27637.log 2024-04-19T01:49:29.072 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27726.log 2024-04-19T01:49:29.072 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.27637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29225.log 2024-04-19T01:49:29.072 INFO:teuthology.orchestra.run.smithi071.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.28963.log.gz 2024-04-19T01:49:29.072 INFO:teuthology.orchestra.run.smithi071.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.30921.log.gz 2024-04-19T01:49:29.072 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.27726.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.29258.log.gz 2024-04-19T01:49:29.072 INFO:teuthology.orchestra.run.smithi071.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.27637.log.gz 2024-04-19T01:49:29.072 INFO:teuthology.orchestra.run.smithi071.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.27726.log.gz 2024-04-19T01:49:29.072 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28820.log 2024-04-19T01:49:29.072 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28236.log 2024-04-19T01:49:29.072 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31392.log 2024-04-19T01:49:29.072 INFO:teuthology.orchestra.run.smithi071.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29225.log.gz 2024-04-19T01:49:29.072 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31704.log 2024-04-19T01:49:29.072 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30311.log 2024-04-19T01:49:29.072 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30747.log 2024-04-19T01:49:29.072 INFO:teuthology.orchestra.run.smithi071.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.28820.log.gz 2024-04-19T01:49:29.073 INFO:teuthology.orchestra.run.smithi071.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.28236.log.gz 2024-04-19T01:49:29.073 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30311.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.31392.log.gz 2024-04-19T01:49:29.073 INFO:teuthology.orchestra.run.smithi071.stderr: 89.7% 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31704.log.gz -- replaced with /var/log/ceph/ceph-client.admin.30311.log.gz 2024-04-19T01:49:29.073 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:49:29.073 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31530.log 2024-04-19T01:49:29.073 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28364.log 2024-04-19T01:49:29.073 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28458.log 2024-04-19T01:49:29.073 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28364.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-19T01:49:29.073 INFO:teuthology.orchestra.run.smithi071.stderr: 89.3%/var/log/ceph/ceph-client.admin.28458.log: -- replaced with /var/log/ceph/ceph-client.admin.30747.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31617.log 2024-04-19T01:49:29.073 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:49:29.073 INFO:teuthology.orchestra.run.smithi071.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.28364.log.gz 2024-04-19T01:49:29.073 INFO:teuthology.orchestra.run.smithi071.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.31530.log.gz 2024-04-19T01:49:29.073 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29084.log 2024-04-19T01:49:29.073 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31617.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.28458.log.gz 2024-04-19T01:49:29.073 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27817.log 2024-04-19T01:49:29.073 INFO:teuthology.orchestra.run.smithi071.stderr: 88.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-19T01:49:29.074 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31182.log 2024-04-19T01:49:29.074 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.27817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29519.log 2024-04-19T01:49:29.074 INFO:teuthology.orchestra.run.smithi071.stderr: 89.5%/var/log/ceph/ceph-client.admin.31182.log: -- replaced with /var/log/ceph/ceph-client.admin.31617.log.gz 2024-04-19T01:49:29.074 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30485.log 2024-04-19T01:49:29.074 INFO:teuthology.orchestra.run.smithi071.stderr: 89.3%gzip/var/log/ceph/ceph-client.admin.29519.log: -- replaced with /var/log/ceph/ceph-client.admin.29084.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.29086.log 2024-04-19T01:49:29.074 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:49:29.074 INFO:teuthology.orchestra.run.smithi071.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.27817.log.gz 2024-04-19T01:49:29.074 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30485.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31182.log.gz 2024-04-19T01:49:29.074 INFO:teuthology.orchestra.run.smithi071.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.29519.log.gz 2024-04-19T01:49:29.074 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27964.log 2024-04-19T01:49:29.074 INFO:teuthology.orchestra.run.smithi071.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.30485.log.gz 2024-04-19T01:49:29.074 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30835.log 2024-04-19T01:49:29.074 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.27964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31356.log 2024-04-19T01:49:29.074 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30835.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.27964.log.gz 2024-04-19T01:49:29.074 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29486.log 2024-04-19T01:49:29.075 INFO:teuthology.orchestra.run.smithi071.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.29086.log.gz 2024-04-19T01:49:29.075 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31651.log 2024-04-19T01:49:29.075 INFO:teuthology.orchestra.run.smithi071.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.30835.log.gz 2024-04-19T01:49:29.075 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29867.log 2024-04-19T01:49:29.075 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23967.log 2024-04-19T01:49:29.075 INFO:teuthology.orchestra.run.smithi071.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29486.log.gz 89.4% 2024-04-19T01:49:29.075 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31356.log.gz/var/log/ceph/ceph-client.admin.29867.log: 2024-04-19T01:49:29.075 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28998.log 2024-04-19T01:49:29.075 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28328.log 2024-04-19T01:49:29.075 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.23967.log: /var/log/ceph/ceph-client.admin.28998.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.31651.log.gz 2024-04-19T01:49:29.075 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28784.log 2024-04-19T01:49:29.075 INFO:teuthology.orchestra.run.smithi071.stderr: 87.7% 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29867.log.gz 2024-04-19T01:49:29.075 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.23967.log.gz 2024-04-19T01:49:29.075 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30869.log 2024-04-19T01:49:29.075 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28328.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28998.log.gz 2024-04-19T01:49:29.076 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31618.log 2024-04-19T01:49:29.076 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30869.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28328.log.gz 2024-04-19T01:49:29.076 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31531.log 2024-04-19T01:49:29.076 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31270.log 2024-04-19T01:49:29.076 INFO:teuthology.orchestra.run.smithi071.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.30869.log.gz 2024-04-19T01:49:29.076 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31531.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.28784.log.gz 2024-04-19T01:49:29.076 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28909.log 2024-04-19T01:49:29.076 INFO:teuthology.orchestra.run.smithi071.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31618.log.gz 2024-04-19T01:49:29.076 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28457.log 2024-04-19T01:49:29.076 INFO:teuthology.orchestra.run.smithi071.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.31531.log.gz 2024-04-19T01:49:29.076 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31270.log: /var/log/ceph/ceph-client.admin.28909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27690.log 2024-04-19T01:49:29.076 INFO:teuthology.orchestra.run.smithi071.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31270.log.gz 2024-04-19T01:49:29.076 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28363.log 2024-04-19T01:49:29.076 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.27690.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.28909.log.gz 2024-04-19T01:49:29.076 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29920.log 2024-04-19T01:49:29.076 INFO:teuthology.orchestra.run.smithi071.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28457.log.gz 2024-04-19T01:49:29.077 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.28363.log: /var/log/ceph/ceph-client.admin.31218.log 2024-04-19T01:49:29.077 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28054.log 2024-04-19T01:49:29.077 INFO:teuthology.orchestra.run.smithi071.stderr: 89.1%/var/log/ceph/ceph-client.admin.31218.log: -- replaced with /var/log/ceph/ceph-client.admin.27690.log.gz 2024-04-19T01:49:29.077 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29259.log 2024-04-19T01:49:29.077 INFO:teuthology.orchestra.run.smithi071.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.28363.log.gz 2024-04-19T01:49:29.077 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28272.log 2024-04-19T01:49:29.077 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28054.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.29920.log.gz 2024-04-19T01:49:29.077 INFO:teuthology.orchestra.run.smithi071.stderr: 90.2%/var/log/ceph/ceph-client.admin.29259.log: -- replaced with /var/log/ceph/ceph-client.admin.31218.log.gz 2024-04-19T01:49:29.077 INFO:teuthology.orchestra.run.smithi071.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.29259.log.gz 2024-04-19T01:49:29.077 INFO:teuthology.orchestra.run.smithi071.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.28054.log.gz 2024-04-19T01:49:29.077 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30660.log 2024-04-19T01:49:29.077 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30661.log 2024-04-19T01:49:29.077 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28547.log 2024-04-19T01:49:29.077 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30486.log 2024-04-19T01:49:29.077 INFO:teuthology.orchestra.run.smithi071.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.28272.log.gz 2024-04-19T01:49:29.077 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28237.log 2024-04-19T01:49:29.078 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28547.log: /var/log/ceph/ceph-client.admin.30486.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.30660.log.gz 88.3% 2024-04-19T01:49:29.078 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30661.log.gz 2024-04-19T01:49:29.078 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28874.log 2024-04-19T01:49:29.078 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29868.log 2024-04-19T01:49:29.078 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28237.log: 88.8%/var/log/ceph/ceph-client.admin.28874.log: -- replaced with /var/log/ceph/ceph-client.admin.30486.log.gz 89.4% -- replaced with /var/log/ceph/ceph-client.admin.28547.log.gz 2024-04-19T01:49:29.078 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:49:29.078 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31827.log 2024-04-19T01:49:29.078 INFO:teuthology.orchestra.run.smithi071.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28237.log.gz 2024-04-19T01:49:29.078 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29868.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.28874.log.gz 2024-04-19T01:49:29.078 INFO:teuthology.orchestra.run.smithi071.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29868.log.gz 2024-04-19T01:49:29.078 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28000.log 2024-04-19T01:49:29.078 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30781.log 2024-04-19T01:49:29.078 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28640.log 2024-04-19T01:49:29.078 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27544.log 2024-04-19T01:49:29.078 INFO:teuthology.orchestra.run.smithi071.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.31827.log.gz 2024-04-19T01:49:29.079 INFO:teuthology.orchestra.run.smithi071.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.28000.log.gz 2024-04-19T01:49:29.079 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27999.log 2024-04-19T01:49:29.079 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.27544.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-19T01:49:29.079 INFO:teuthology.orchestra.run.smithi071.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.30781.log.gz/var/log/ceph/ceph-client.admin.27999.log: 2024-04-19T01:49:29.079 INFO:teuthology.orchestra.run.smithi071.stderr:gzip 90.3% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.28640.log.gz 2024-04-19T01:49:29.079 INFO:teuthology.orchestra.run.smithi071.stderr: -- /var/log/ceph/ceph-client.admin.28638.log 2024-04-19T01:49:29.092 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27963.log 2024-04-19T01:49:29.092 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.28638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23888.log 2024-04-19T01:49:29.092 INFO:teuthology.orchestra.run.smithi071.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.27544.log.gz 2024-04-19T01:49:29.092 INFO:teuthology.orchestra.run.smithi071.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.27999.log.gz 2024-04-19T01:49:29.092 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.27963.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.28638.log.gz 2024-04-19T01:49:29.092 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28785.log 2024-04-19T01:49:29.092 INFO:teuthology.orchestra.run.smithi071.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.27963.log.gz 2024-04-19T01:49:29.092 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.23888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28418.log 2024-04-19T01:49:29.092 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30042.log 2024-04-19T01:49:29.092 INFO:teuthology.orchestra.run.smithi071.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.23888.log.gz 2024-04-19T01:49:29.093 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23864.log 2024-04-19T01:49:29.093 INFO:teuthology.orchestra.run.smithi071.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28785.log.gz 2024-04-19T01:49:29.093 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31357.log 2024-04-19T01:49:29.093 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.23864.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.30042.log.gz 2024-04-19T01:49:29.093 INFO:teuthology.orchestra.run.smithi071.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.28418.log.gz 2024-04-19T01:49:29.093 INFO:teuthology.orchestra.run.smithi071.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.23864.log.gz 2024-04-19T01:49:29.093 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30573.log 2024-04-19T01:49:29.093 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31357.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-19T01:49:29.093 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30573.log: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31183.log 2024-04-19T01:49:29.093 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31357.log.gz 2024-04-19T01:49:29.093 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29173.log 2024-04-19T01:49:29.093 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30520.log 2024-04-19T01:49:29.093 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31183.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.30573.log.gz 2024-04-19T01:49:29.093 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29173.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.31183.log.gz 2024-04-19T01:49:29.093 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31043.log 2024-04-19T01:49:29.093 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30344.log 2024-04-19T01:49:29.094 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31043.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.29173.log.gz 2024-04-19T01:49:29.094 INFO:teuthology.orchestra.run.smithi071.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.30520.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27819.log 2024-04-19T01:49:29.094 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:49:29.094 INFO:teuthology.orchestra.run.smithi071.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31043.log.gz 2024-04-19T01:49:29.094 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31269.log 2024-04-19T01:49:29.094 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.27819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29660.log 2024-04-19T01:49:29.094 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31269.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.27819.log.gz 2024-04-19T01:49:29.094 INFO:teuthology.orchestra.run.smithi071.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.30344.log.gz 2024-04-19T01:49:29.094 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28092.log 2024-04-19T01:49:29.094 INFO:teuthology.orchestra.run.smithi071.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.31269.log.gz 2024-04-19T01:49:29.094 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29747.log 2024-04-19T01:49:29.094 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28182.log 2024-04-19T01:49:29.094 INFO:teuthology.orchestra.run.smithi071.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29660.log.gz 2024-04-19T01:49:29.094 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29747.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.29747.log.gz 2024-04-19T01:49:29.095 INFO:teuthology.orchestra.run.smithi071.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.28092.log.gz 2024-04-19T01:49:29.095 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27873.log 2024-04-19T01:49:29.095 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29433.log 2024-04-19T01:49:29.095 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.27873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30215.log 2024-04-19T01:49:29.095 INFO:teuthology.orchestra.run.smithi071.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.28182.log.gz 2024-04-19T01:49:29.095 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29433.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.27873.log.gz 2024-04-19T01:49:29.095 INFO:teuthology.orchestra.run.smithi071.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.29433.log.gz 2024-04-19T01:49:29.095 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27782.log 2024-04-19T01:49:29.095 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28999.log 2024-04-19T01:49:29.095 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.27782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30182.log 2024-04-19T01:49:29.095 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28999.log: 88.2% 89.4% -- replaced with /var/log/ceph/ceph-client.admin.27782.log.gz 2024-04-19T01:49:29.095 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30215.log.gz 2024-04-19T01:49:29.095 INFO:teuthology.orchestra.run.smithi071.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.28999.log.gz 2024-04-19T01:49:29.095 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28548.log 2024-04-19T01:49:29.095 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23936.log 2024-04-19T01:49:29.095 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28183.log 2024-04-19T01:49:29.096 INFO:teuthology.orchestra.run.smithi071.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30182.log.gz 2024-04-19T01:49:29.096 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.23936.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.28548.log.gz 2024-04-19T01:49:29.096 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29345.log 2024-04-19T01:49:29.096 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.28183.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.23936.log.gz 2024-04-19T01:49:29.096 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30397.log 2024-04-19T01:49:29.096 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29345.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.28183.log.gz 2024-04-19T01:49:29.096 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31791.log 2024-04-19T01:49:29.096 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31479.log 2024-04-19T01:49:29.096 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30397.log: /var/log/ceph/ceph-client.admin.31791.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29345.log.gz 2024-04-19T01:49:29.096 INFO:teuthology.orchestra.run.smithi071.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.30397.log.gz 2024-04-19T01:49:29.096 INFO:teuthology.orchestra.run.smithi071.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31791.log.gz 2024-04-19T01:49:29.096 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30695.log 2024-04-19T01:49:29.096 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.31479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29695.log 2024-04-19T01:49:29.096 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30129.log 2024-04-19T01:49:29.096 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.29695.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.31479.log.gz 2024-04-19T01:49:29.097 INFO:teuthology.orchestra.run.smithi071.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.30695.log.gz 2024-04-19T01:49:29.097 INFO:teuthology.orchestra.run.smithi071.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.29695.log.gz 2024-04-19T01:49:29.097 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30782.log 2024-04-19T01:49:29.097 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.30129.log: /var/log/ceph/ceph-client.admin.30782.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.30129.log.gz 2024-04-19T01:49:29.097 INFO:teuthology.orchestra.run.smithi071.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30782.log.gz 2024-04-19T01:49:29.108 INFO:teuthology.orchestra.run.smithi071.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-19T01:49:29.191 INFO:teuthology.orchestra.run.smithi062.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-19T01:49:29.323 INFO:teuthology.orchestra.run.smithi071.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-19T01:49:29.332 INFO:teuthology.orchestra.run.smithi062.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-19T01:49:31.208 INFO:teuthology.orchestra.run.smithi062.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-19T01:49:31.223 INFO:teuthology.orchestra.run.smithi071.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-19T01:49:31.258 INFO:teuthology.orchestra.run.smithi062.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-19T01:49:31.271 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-04-19T01:49:31.466 INFO:teuthology.orchestra.run.smithi071.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-19T01:49:31.568 INFO:teuthology.orchestra.run.smithi071.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-19T01:49:31.569 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-19T01:49:31.570 INFO:teuthology.orchestra.run.smithi071.stderr:real 0m2.634s 2024-04-19T01:49:31.570 INFO:teuthology.orchestra.run.smithi071.stderr:user 0m7.747s 2024-04-19T01:49:31.570 INFO:teuthology.orchestra.run.smithi071.stderr:sys 0m0.310s 2024-04-19T01:49:31.658 INFO:teuthology.orchestra.run.smithi062.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-19T01:49:31.658 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-19T01:49:31.659 INFO:teuthology.orchestra.run.smithi062.stderr:real 0m2.756s 2024-04-19T01:49:31.659 INFO:teuthology.orchestra.run.smithi062.stderr:user 0m7.886s 2024-04-19T01:49:31.659 INFO:teuthology.orchestra.run.smithi062.stderr:sys 0m0.274s 2024-04-19T01:49:31.660 INFO:tasks.ceph:Archiving logs... 2024-04-19T01:49:31.660 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663309/remote/smithi062/log 2024-04-19T01:49:31.661 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-19T01:49:33.021 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663309/remote/smithi071/log 2024-04-19T01:49:33.022 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-19T01:49:34.369 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-19T01:49:34.384 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-19T01:49:34.384 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-19T01:49:34.387 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-19T01:49:34.449 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-19T01:49:34.460 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-19T01:49:34.460 DEBUG:teuthology.orchestra.run.smithi062:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-19T01:49:34.462 DEBUG:teuthology.orchestra.run.smithi071:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-19T01:49:34.474 INFO:teuthology.orchestra.run.smithi071.stdout: remote refid st t when poll reach delay offset jitter 2024-04-19T01:49:34.474 INFO:teuthology.orchestra.run.smithi071.stdout:============================================================================== 2024-04-19T01:49:34.474 INFO:teuthology.orchestra.run.smithi071.stdout:*hv01.front.sepi 67.205.162.81 3 u 7 64 377 0.088 -1.347 1.555 2024-04-19T01:49:34.474 INFO:teuthology.orchestra.run.smithi071.stdout:+hv02.front.sepi 63.231.80.2 3 u 7 64 377 0.106 -1.012 1.157 2024-04-19T01:49:34.474 INFO:teuthology.orchestra.run.smithi071.stdout:+hv03.front.sepi 74.6.168.72 3 u 1 64 377 0.087 -2.131 0.918 2024-04-19T01:49:34.474 INFO:teuthology.orchestra.run.smithi071.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:49:34.475 INFO:teuthology.orchestra.run.smithi062.stdout: remote refid st t when poll reach delay offset jitter 2024-04-19T01:49:34.475 INFO:teuthology.orchestra.run.smithi062.stdout:============================================================================== 2024-04-19T01:49:34.476 INFO:teuthology.orchestra.run.smithi062.stdout:*hv01.front.sepi 67.205.162.81 3 u 5 64 377 0.089 -2.829 0.655 2024-04-19T01:49:34.476 INFO:teuthology.orchestra.run.smithi062.stdout:+hv02.front.sepi 63.231.80.2 3 u 2 64 377 0.083 -0.349 0.721 2024-04-19T01:49:34.476 INFO:teuthology.orchestra.run.smithi062.stdout:+hv03.front.sepi 74.6.168.72 3 u 3 64 377 0.078 -2.163 0.671 2024-04-19T01:49:34.476 INFO:teuthology.orchestra.run.smithi062.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T01:49:34.476 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-19T01:49:34.487 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-19T01:49:34.489 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-19T01:49:34.500 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-19T01:49:34.511 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-19T01:49:34.523 INFO:teuthology.task.internal:Duration was 1033.363622 seconds 2024-04-19T01:49:34.524 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-19T01:49:34.535 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-19T01:49:34.536 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-19T01:49:34.539 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-19T01:49:34.577 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-19T01:49:34.578 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi062.front.sepia.ceph.com 2024-04-19T01:49:34.578 DEBUG:teuthology.orchestra.run.smithi062:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-19T01:49:34.634 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi071.front.sepia.ceph.com 2024-04-19T01:49:34.634 DEBUG:teuthology.orchestra.run.smithi071:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-19T01:49:34.645 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-19T01:49:34.645 DEBUG:teuthology.orchestra.run.smithi062:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-19T01:49:34.677 DEBUG:teuthology.orchestra.run.smithi071:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-19T01:49:34.712 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-19T01:49:34.713 DEBUG:teuthology.orchestra.run.smithi062:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-19T01:49:34.745 DEBUG:teuthology.orchestra.run.smithi071:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-19T01:49:34.809 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-19T01:49:34.820 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-19T01:49:34.820 DEBUG:teuthology.orchestra.run.smithi062:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-19T01:49:34.833 DEBUG:teuthology.orchestra.run.smithi071:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-19T01:49:34.844 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-19T01:49:34.856 DEBUG:teuthology.orchestra.run.smithi062:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-19T01:49:34.882 DEBUG:teuthology.orchestra.run.smithi071:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-19T01:49:34.890 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern = core 2024-04-19T01:49:34.897 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = core 2024-04-19T01:49:35.003 DEBUG:teuthology.orchestra.run.smithi062:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-19T01:49:35.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:49:35.009 DEBUG:teuthology.orchestra.run.smithi071:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-19T01:49:35.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:49:35.052 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-19T01:49:35.064 INFO:teuthology.task.internal:Transferring archived files... 2024-04-19T01:49:35.065 DEBUG:teuthology.misc:Transferring archived files from smithi062:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663309/remote/smithi062 2024-04-19T01:49:35.065 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-19T01:49:35.089 DEBUG:teuthology.misc:Transferring archived files from smithi071:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663309/remote/smithi071 2024-04-19T01:49:35.090 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-19T01:49:35.117 INFO:teuthology.task.internal:Removing archive directory... 2024-04-19T01:49:35.118 DEBUG:teuthology.orchestra.run.smithi062:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-19T01:49:35.129 DEBUG:teuthology.orchestra.run.smithi071:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-19T01:49:35.160 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-19T01:49:35.171 INFO:teuthology.task.internal:Not uploading archives. 2024-04-19T01:49:35.171 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-19T01:49:35.184 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-19T01:49:35.184 DEBUG:teuthology.orchestra.run.smithi062:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-19T01:49:35.187 DEBUG:teuthology.orchestra.run.smithi071:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-19T01:49:35.191 INFO:teuthology.orchestra.run.smithi062.stdout: 658640 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 19 01:49 /home/ubuntu/cephtest 2024-04-19T01:49:35.208 INFO:teuthology.orchestra.run.smithi071.stdout: 658467 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 19 01:49 /home/ubuntu/cephtest 2024-04-19T01:49:35.209 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-19T01:49:35.221 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-19T01:49:35.247 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/pool-create-delete} duration: 1033.363621711731 flavor: default owner: scheduled_yuriw@teuthology success: true 2024-04-19T01:49:35.247 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-19T01:49:35.329 INFO:teuthology.run:pass